Ethernet fabric protection in a disaggregated OTN switching system

ABSTRACT

Methods and systems for Ethernet fabric protection in a disaggregated OTN switching system that include PIU modules each having multiple ports for OTN to Ethernet transceiving and an Ethernet fabric as a switching core are disclosed. An OTN over Ethernet module in each of the PIU modules may enable various OTN functionality to be realized using the Ethernet fabric which may include multiple Ethernet switches. A first PIU module may detect a fault condition on an Ethernet fabric plane of the Ethernet fabric. In response to the detection, the OTN switching system may transmit the fault condition to other PIU modules to redirect optical data unit traffic away from the fault on the Ethernet fabric plane.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Provisional Application No.62/325,723 filed Apr. 21, 2016, entitled “DISAGGREGATED OPTICALTRANSPORT NETWORK SWITCHING SYSTEM”.

BACKGROUND Field of the Disclosure

The present disclosure relates generally to optical communicationnetworks and, more particularly, to Ethernet fabric protection in adisaggregated optical transport network switching system.

Description of the Related Art

Telecommunication, cable television and data communication systems useoptical transport networks (OTN) to rapidly convey large amounts ofinformation between remote points. In an OTN, information is conveyed inthe form of optical signals through optical fibers, where multiplesub-channels may be carried within an optical signal. OTNs may alsoinclude various network elements, such as amplifiers, dispersioncompensators, multiplexer/demultiplexer filters, wavelength selectiveswitches, optical switches, couplers, etc. configured to perform variousoperations within the network.

In particular, OTNs may be reconfigured to transmit different individualchannels using, for example, optical add-drop multiplexers (OADMs). Inthis manner, individual channels (e.g., wavelengths) may be added ordropped at various points along an optical network, enabling a varietyof network configurations and topologies.

Furthermore, typically, an optical transport network (OTN) switch isused to centrally perform electrical switching of the sub-channelscarried within an optical signal to different destinations.

SUMMARY

Methods and systems for Ethernet fabric protection in a disaggregatedoptical transport network (OTN) switching system that include usingplug-in universal (PIU) modules each having multiple ports for OTN toEthernet transceiving and an Ethernet fabric as a switching core aredisclosed. An OTN over Ethernet module in each of the PIU modules mayenable various OTN functionality to be realized using the Ethernetfabric which may include multiple Ethernet switches. A first PIU modulemay detect a fault condition on an Ethernet fabric plane of the Ethernetfabric. In response to the detection, the OTN switching system maytransmit the fault condition to other PIU modules to redirect opticaldata unit (ODU) traffic away from the fault on the Ethernet fabricplane.

In one aspect, a disclosed method for Ethernet fabric protection in anOTN switching system may include, in an OTN switch that may include anEthernet fabric having a number M of Ethernet fabric planes, each of theM Ethernet fabric planes may include a corresponding Ethernet switch ofM Ethernet switches. The OTN switch may also include a plurality of PIUmodules each having M PIU ports including a first PIU module, where anith PIU port of each of the plurality of PIU modules may be connected tothe ith Ethernet switch of the ith Ethernet fabric plane of the Ethernetfabric. The method may also include assigning a variable i having avalue ranging from 1 to M to denote the ith Ethernet fabric plane of theM Ethernet fabric planes, the ith Ethernet switch of the M Ethernetswitches, and the ith PIU port of the M PIU ports, where M is greaterthan one. The method may further include detecting, by the first PIUmodule, a fault condition associated with the ith PIU port of the firstPIU module on the ith Ethernet fabric plane. The method may also includetransmitting the fault condition to stop transmission of ODU trafficfrom the plurality of PIU modules to the ith PIU port of the first PIUmodule.

In any of the disclosed embodiments of the method, the method may alsoinclude after transmitting the fault condition to stop the transmissionof the ODU traffic from the plurality of PIU modules to the ith PIU portof the first PIU module, stopping the transmission of the ODU trafficfrom the plurality of PIU modules to the ith PIU port of the first PIUmodule.

In any of the disclosed embodiments of the method, the method mayfurther include detecting, by a second PIU module of the plurality ofPIU modules, a second fault condition associated with the ith PIU portof the second PIU module on the ith Ethernet fabric plane. The methodmay also include transmitting the second fault condition to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the second PIU module.

In any of the disclosed embodiments of the method, where transmittingthe fault condition and transmitting the second fault condition may betransmitted in the same transmission.

In any of the disclosed embodiments of the method, the method may alsoinclude after receiving the fault condition and a second fault conditionassociated with the ith PIU port of a second PIU module of the pluralityof PIU modules on the ith Ethernet fabric plane to stop the transmissionof the ODU traffic from the plurality of PIU modules to the ith PIU portof the second PIU module, stopping the transmission of the ODU trafficfrom the plurality of PIU modules to the ith PIU port of the first PIUmodule and the ith PIU port of the second PIU module.

In any of the disclosed embodiments of the method, the method mayfurther include prior to transmitting the fault condition to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the first PIU module and transmitting a second faultcondition associated with the ith PIU port of a second PIU module of theplurality of PIU modules on the ith Ethernet fabric plane to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the second PIU module, stopping the transmission of theODU traffic from the M PIU ports of the first PIU module and the M PIUports of the second PIU module. The method may also include afterexpiration of a delay associated with the first PIU module and thesecond PIU module, transmitting the second fault condition to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the second PIU module.

In any of the disclosed embodiments of the method, the method mayfurther include after expiration of a second delay associated with thefirst PIU module and the second PIU module, transmitting the ODU trafficfrom the M PIU ports of the first PIU module other than the ith PIU portof the first PIU module and transmitting the ODU traffic from the M PIUports of the second PIU module other than the ith PIU port of the secondPIU module, where the expiration of the second delay is after theexpiration of the delay.

In any of the disclosed embodiments of the method, the method mayfurther include receiving a third fault condition associated with thejth PIU port of a third PIU module on the ith Ethernet fabric plane tostop the transmission of the ODU traffic from the plurality of PIUmodules to the jth PIU port of the third PIU module. The method may alsoinclude after expiration of a second delay associated with the first PIUmodule and the second PIU module, transmitting the ODU traffic from theM PIU ports of the first PIU module, other than the ith PIU port of thefirst PIU module to the plurality of PIU modules other than the ith PIUport of the first PIU module, the ith PIU port of the second PIU module,and the jth PIU port of the third PIU module. The method may alsoinclude after expiration of the second delay, transmitting the ODUtraffic from the M PIU ports of the second PIU module, other than theith PIU port of the second PIU module, to the plurality of PIU modulesother than the ith PIU port of the first PIU module, the ith PIU port ofthe second PIU module, and the jth PIU port of the third PIU module,where the expiration of the second delay is after the expiration of thedelay.

In any of the disclosed embodiments of the method, the method mayfurther include prior to transmitting the fault condition to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the first PIU module and transmitting a second faultcondition associated with the ith PIU port of a second PIU module of theplurality of PIU modules on the ith Ethernet fabric plane to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the second PIU module, stopping the transmission of theODU traffic from the M PIU ports of the first PIU module and the M PIUports of the second PIU module. The method may also include receiving athird fault condition associated with the jth PIU port of a third PIUmodule on the ith Ethernet fabric plane to stop the transmission of theODU traffic from the plurality of PIU modules to the jth PIU port of thethird PIU module. The method may further include after expiration of asecond delay associated with the first PIU module and the second PIUmodule, transmitting the second fault condition to stop the transmissionof the ODU traffic from the plurality of PIU modules to the ith PIU portof the second PIU module. The method may further include afterexpiration of the second delay, transmitting the third fault conditionto stop the transmission of the ODU traffic from the plurality of PIUmodules to the jth PIU port of the third PIU module.

In any of the disclosed embodiments of the method, the method mayfurther include after expiration of a third delay associated with thefirst PIU module and the second PIU module, transmitting the ODU trafficfrom the M PIU ports of the first PIU module, other than the ith PIUport of the first PIU module. The method may also include afterexpiration of the third delay, transmitting the ODU traffic from the MPIU ports of the second PIU module, other than the ith PIU port of thesecond PIU module, to the plurality of PIU modules other than the ithPIU port of the first PIU module, the ith PIU port of the second PIUmodule, and the jth PIU port of the third PIU module, where theexpiration of the third delay is after the expiration of the seconddelay.

In any of the disclosed embodiments of the method, the method mayfurther include after receiving the fault condition and a second faultcondition associated with the ith PIU port of a second PIU module of theplurality of PIU modules on the ith Ethernet fabric plane to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the second PIU module, stopping the transmission of theODU traffic from the M PIU ports of a third PIU module of the pluralityof PIU modules. The method may also include after expiration of a delayassociated with the third PIU module, transmitting the ODU traffic fromthe M PIU ports of the third PIU module to the plurality of PIU modulesother than the ith PIU port of the first PIU module and the ith PIU portof the second PIU module.

In any of the disclosed embodiments of the method, the method mayfurther include detecting a second fault condition of the ith Ethernetfabric plane. The method may also include transmitting the ODU trafficfrom the plurality of PIU modules to the other Ethernet fabric planes.

In another aspect, a disclosed OTN switching system for Ethernet fabricprotection may include an OTN switch including an Ethernet fabric havinga number M of Ethernet fabric planes, each of the M Ethernet fabricplanes include a corresponding Ethernet switch of M Ethernet switches.The OTN switch may also include a plurality of PIU modules each having MPIU ports including a first PIU module, where an ith PIU port of each ofthe plurality of PIU modules is connected to the ith Ethernet switch ofthe ith Ethernet fabric plane of the Ethernet fabric, and where avariable i having a value ranging from 1 to M to denote the ith Ethernetfabric plane of the M Ethernet fabric planes, the ith Ethernet switch ofthe M Ethernet switches, and the ith PIU port of the M PIU ports, whereM is greater than one. The first PIU module may detect a fault conditionassociated with the ith PIU port of the first PIU module on the ithEthernet fabric plane. The OTN switch may transmit the fault conditionto stop transmission of ODU traffic from the plurality of PIU modules tothe ith PIU port of the first PIU module.

In any of the disclosed embodiments of the OTN switching system, the OTNswitching system may further include a second PIU module of theplurality of PIU modules that may detect a second fault conditionassociated with the ith PIU port of the second PIU module on the ithEthernet fabric plane. The OTN switch may transmit the second faultcondition to stop the transmission of the ODU traffic from the pluralityof PIU modules to the ith PIU port of the second PIU module.

In any of the disclosed embodiments of the OTN switching system, the OTNswitching system may further include the OTN switch, after receiving thefault condition and a second fault condition associated with the ith PIUport of a second PIU module of the plurality of PIU modules on the ithEthernet fabric plane, may stop the transmission of the ODU traffic fromthe plurality of PIU modules to the ith PIU port of the second PIUmodule. The OTN switch may also, after receiving the fault condition andthe second fault condition, stop the transmission of the ODU trafficfrom the plurality of PIU modules to the ith PIU port of the first PIUmodule and the ith PIU port of the second PIU module.

In any of the disclosed embodiments of the OTN switching system, the OTNswitching system may further include the OTN switch, prior to thetransmission of the fault condition to stop the transmission of the ODUtraffic from the plurality of PIU modules to the ith PIU port of thefirst PIU module and the transmission of a second fault conditionassociated with the ith PIU port of a second PIU module of the pluralityof PIU modules on the ith Ethernet fabric plane, may stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the second PIU module. The OTN switch, prior to thetransmission of the fault condition and the transmission of the secondfault condition, may stop the transmission of the ODU traffic from the MPIU ports of the first PIU module and the M PIU ports of the second PIUmodule. The OTN switch, after expiration of a delay associated with thefirst PIU module and the second PIU module, may transmit the secondfault condition to stop the transmission of the ODU traffic from theplurality of PIU modules to the ith PIU port of the second PIU module.

In any of the disclosed embodiments of the OTN switching system, the OTNswitching system may further include the OTN switch, after expiration ofa second delay associated with the first PIU module and the second PIUmodule, may transmit the ODU traffic from the M PIU ports of the firstPIU module other than the ith PIU port of the first PIU module. The OTNswitch, after expiration of the second delay, may transmit the ODUtraffic from the M PIU ports of the second PIU module other than the ithPIU port of the second PIU module, where the expiration of the seconddelay is after the expiration of the delay.

In any of the disclosed embodiments of the OTN switching system, the OTNswitching system may further include a third PIU module of the pluralityof PIU modules. The OTN switch, prior to the transmission of the faultcondition to stop the transmission of the ODU traffic from the pluralityof PIU modules to the ith PIU port of the first PIU module and thetransmission of a second fault condition associated with the ith PIUport of a second PIU module of the plurality of PIU modules on the ithEthernet fabric plane to stop the transmission of the ODU traffic fromthe plurality of PIU modules to the ith PIU port of the second PIUmodule, may stop the transmission of the ODU traffic from the M PIUports of the first PIU module and the M PIU ports of the second PIUmodule. The OTN switch may also receive a third fault conditionassociated with the jth PIU port of the third PIU module on the ithEthernet fabric plane to stop the transmission of the ODU traffic fromthe plurality of PIU modules to the jth PIU port of the third PIUmodule. The OTN switch, after expiration of a second delay associatedwith the first PIU module and the second PIU module, may transmit thesecond fault condition to stop the transmission of the ODU traffic fromthe plurality of PIU modules to the ith PIU port of the second PIUmodule. The OTN switch, after expiration of the second delay, may alsotransmit the third fault condition to stop the transmission of the ODUtraffic from the plurality of PIU modules to the jth PIU port of thethird PIU module.

In any of the disclosed embodiments of the OTN switching system, the OTNswitching system may further include the OTN switch, after expiration ofa third delay associated with the first PIU module and the second PIUmodule, may transmit the ODU traffic from the M PIU ports of the firstPIU module, other than the ith PIU port of the first PIU module. The OTNswitch, after expiration of the third delay, may transmit the ODUtraffic from the M PIU ports of the second PIU module, other than theith PIU port of the second PIU module, to the plurality of PIU modulesother than the ith PIU port of the first PIU module, the ith PIU port ofthe second PIU module, and the jth PIU port of the third PIU module,where the expiration of the third delay is after the expiration of thesecond delay.

In any of the disclosed embodiments of the OTN switching system, the OTNswitching system may further include a second PIU module of theplurality of PIU modules. The OTN switching system may also include athird PIU module of the plurality of PIU modules, after receiving thefault condition and a second fault condition associated with the ith PIUport of the second PIU module on the ith Ethernet fabric plane, may stopthe transmission of the ODU traffic from the plurality of PIU modules tothe ith PIU port of the second PIU module. The third PIU module, afterreceiving the fault condition and the second fault condition, may alsostop the transmission of the ODU traffic from the M PIU ports of thethird PIU module. The third PIU module, after expiration of a delayassociated with the third PIU module, may also transmit the ODU trafficfrom the M PIU ports of the third PIU module to the plurality of PIUmodules other than the ith PIU port of the first PIU module and the ithPIU port of the second PIU module.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention and itsfeatures and advantages, reference is now made to the followingdescription, taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a block diagram of selected elements of an embodiment of anoptical transport network (OTN);

FIG. 2 is a block diagram of selected elements of an embodiment of adisaggregated OTN switching system;

FIGS. 3 and 4 are block diagrams of selected elements of an embodimentof an OTN switch network element controller;

FIG. 5 is a block diagram of selected elements of an embodiment of a PIUchassis;

FIGS. 6A, 6B, and 6C are block diagrams of selected elements ofembodiments of PIU modules;

FIGS. 7A and 7B are a block diagrams of selected elements of anembodiment of local OTN switching functionality;

FIG. 8 is a block diagram of an example of optical data unit (ODU)forwarding over an embodiment of an Ethernet fabric;

FIG. 9 is a block diagram of an example of ODU forwarding using acyclical walk sequence over an embodiment of an Ethernet fabric in anOTN switching system;

FIG. 10 is a block diagram of multiple ODU switched connections throughan embodiment of a single PIU module;

FIG. 11 is a block diagram of an embodiment of virtual slots in anEthernet switch fabric;

FIG. 12 is a block diagram of an example of ODU path protection in anembodiment of an OTN switching system;

FIG. 13 is a block diagram of an example of concatenation of ODU pathprotection in an embodiment of an OTN switching system;

FIG. 14 is a block diagram of an example embodiment of an OTN switchingsystem;

FIG. 15 is a block diagram of an example of Ethernet fabric protectionin an embodiment of an OTN switching system; and

FIG. 16 is a flowchart of selected elements of an embodiment of a methodfor Ethernet fabric protection in an OTN switching system.

DESCRIPTION OF PARTICULAR EMBODIMENT(S)

In the following description, details are set forth by way of example tofacilitate discussion of the disclosed subject matter. It should beapparent to a person of ordinary skill in the field, however, that thedisclosed embodiments are exemplary and not exhaustive of all possibleembodiments.

Throughout this disclosure, a hyphenated form of a reference numeralrefers to a specific instance of an element and the un-hyphenated formof the reference numeral refers to the element generically orcollectively. Thus, as an example (not shown in the drawings), device“12-1” refers to an instance of a device class, which may be referred tocollectively as devices “12” and any one of which may be referred togenerically as a device “12”. In the figures and the description, likenumerals are intended to represent like elements.

Turning now to the drawings, FIG. 1 illustrates an example embodiment ofan optical transport network 101, which may represent an opticalcommunication system. Optical transport network 101 may include one ormore optical fibers 106 configured to transport one or more opticalsignals communicated by components of optical transport network 101. Thenetwork elements of optical transport network 101, coupled together byfibers 106, may comprise one or more transmitters 102, one or moremultiplexers (MUX) 104, one or more optical amplifiers 108, one or moreoptical add/drop multiplexers (OADM) 110, one or more demultiplexers(DEMUX) 105, and one or more receivers 112.

Optical transport network 101 may comprise a point-to-point opticalnetwork with terminal nodes, a ring optical network, a mesh opticalnetwork, or any other suitable optical network or combination of opticalnetworks. Optical fibers 106 comprise thin strands of glass capable ofcommunicating the signals over long distances with very low loss.Optical fibers 106 may comprise a suitable type of fiber selected from avariety of different fibers for optical transmission.

Optical transport network 101 may include devices configured to transmitoptical signals over optical fibers 106. Information may be transmittedand received through optical transport network 101 by modulation of oneor more wavelengths of light to encode the information on thewavelength. In optical networking, a wavelength of light may also bereferred to as a channel. Each channel may be configured to carry acertain amount of information through optical transport network 101.

To increase the information capacity and transport capabilities ofoptical transport network 101, multiple signals transmitted at multiplechannels may be combined into a single wideband optical signal. Theprocess of communicating information at multiple channels is referred toin optics as wavelength division multiplexing (WDM). Coarse wavelengthdivision multiplexing (CWDM) refers to the multiplexing of wavelengthsthat are widely spaced having low number of channels, usually greaterthan 20 nm and less than sixteen wavelengths, and dense wavelengthdivision multiplexing (DWDM) refers to the multiplexing of wavelengthsthat are closely spaced having large number of channels, usually lessthan 0.8 nm spacing and greater than forty wavelengths, into a fiber.WDM or other multi-wavelength multiplexing transmission techniques areemployed in optical networks to increase the aggregate bandwidth peroptical fiber. Without WDM, the bandwidth in optical networks may belimited to the bit-rate of solely one wavelength. With more bandwidth,optical networks are capable of transmitting greater amounts ofinformation. Optical transport network 101 may be configured to transmitdisparate channels using WDM or some other suitable multi-channelmultiplexing technique, and to amplify the multi-channel signal.

Optical transport network 101 may include one or more opticaltransmitters (Tx) 102 configured to transmit optical signals throughoptical transport network 101 in specific wavelengths or channels.Transmitters 102 may comprise a system, apparatus or device configuredto convert an electrical signal into an optical signal and transmit theoptical signal. For example, transmitters 102 may each comprise a laserand a modulator to receive electrical signals and modulate theinformation contained in the electrical signals onto a beam of lightproduced by the laser at a particular wavelength, and transmit the beamfor carrying the signal throughout optical transport network 101.

Multiplexer 104 may be coupled to transmitters 102 and may be a system,apparatus or device configured to combine the signals transmitted bytransmitters 102, e.g., at respective individual wavelengths, into a WDMsignal.

Optical amplifiers 108 may amplify the multi-channeled signals withinoptical transport network 101. Optical amplifiers 108 may be positionedbefore and after certain lengths of fiber 106. Optical amplifiers 108may comprise a system, apparatus, or device configured to amplifyoptical signals. For example, optical amplifiers 108 may comprise anoptical repeater that amplifies the optical signal. This amplificationmay be performed with opto-electrical (O-E) or electro-optical (E-O)conversion. In some embodiments, optical amplifiers 108 may comprise anoptical fiber doped with a rare-earth element to form a doped fiberamplification element. When a signal passes through the fiber, externalenergy may be applied in the form of a pump signal to excite the atomsof the doped portion of the optical fiber, which increases the intensityof the optical signal. As an example, optical amplifiers 108 maycomprise an erbium-doped fiber amplifier (EDFA).

OADMs 110 may be coupled to optical transport network 101 via fibers106. OADMs 110 comprise an add/drop module, which may include a system,apparatus or device configured to add or drop optical signals (i.e., atindividual wavelengths) from fibers 106. After passing through an OADM110, an optical signal may travel along fibers 106 directly to adestination, or the signal may be passed through one or more additionalOADMs 110 and optical amplifiers 108 before reaching a destination.

In certain embodiments of optical transport network 101, OADM 110 mayrepresent a reconfigurable OADM (ROADM) that is capable of adding ordropping individual or multiple wavelengths of a WDM signal. Theindividual or multiple wavelengths may be added or dropped in theoptical domain, for example, using a wavelength selective switch (WSS)(not shown) that may be included in a ROADM.

As shown in FIG. 1, optical transport network 101 may also include oneor more demultiplexers 105 at one or more destinations of network 101.Demultiplexer 105 may comprise a system apparatus or device that acts asa demultiplexer by splitting a single composite WDM signal intoindividual channels at respective wavelengths. For example, opticaltransport network 101 may transmit and carry a forty (40) channel DWDMsignal. Demultiplexer 105 may divide the single, forty channel DWDMsignal into forty separate signals according to the forty differentchannels.

In FIG. 1, optical transport network 101 may also include receivers 112coupled to demultiplexer 105. Each receiver 112 may be configured toreceive optical signals transmitted at a particular wavelength orchannel, and may process the optical signals to obtain (e.g.,demodulate) the information (i.e., data) that the optical signalscontain. Accordingly, network 101 may include at least one receiver 112for every channel of the network.

Optical networks, such as optical transport network 101 in FIG. 1, mayemploy modulation techniques to convey information in the opticalsignals over the optical fibers. Such modulation schemes may includephase-shift keying (PSK), frequency-shift keying (FSK), amplitude-shiftkeying (ASK), and quadrature amplitude modulation (QAM), among otherexamples of modulation techniques. In PSK, the information carried bythe optical signal may be conveyed by modulating the phase of areference signal, also known as a carrier wave, or simply, a carrier.The information may be conveyed by modulating the phase of the signalitself using two-level or binary phase-shift keying (BPSK), four-levelor quadrature phase-shift keying (QPSK), multi-level phase-shift keying(M-PSK) and differential phase-shift keying (DPSK). In QAM, theinformation carried by the optical signal may be conveyed by modulatingboth the amplitude and phase of the carrier wave. PSK may be considereda subset of QAM, wherein the amplitude of the carrier waves ismaintained as a constant. Additionally, polarization divisionmultiplexing (PDM) technology may enable achieving a greater bit ratefor information transmission. PDM transmission comprises modulatinginformation onto various polarization components of an optical signalassociated with a channel. The polarization of an optical signal mayrefer to the direction of the oscillations of the optical signal. Theterm “polarization” may generally refer to the path traced out by thetip of the electric field vector at a point in space, which isperpendicular to the propagation direction of the optical signal.

In an optical network, such as optical transport network 101 in FIG. 1,it is typical to refer to a management plane, a control plane, and atransport plane (sometimes called the physical layer). A centralmanagement host (not shown) may reside in the management plane and mayconfigure and supervise the components of the control plane. Themanagement plane includes ultimate control over all transport plane andcontrol plane entities (e.g., network elements). As an example, themanagement plane may consist of a central processing center (e.g., thecentral management host), including one or more processing resources,data storage components, etc. The management plane may be in electricalcommunication with the elements of the control plane and may also be inelectrical communication with one or more network elements of thetransport plane. The management plane may perform management functionsfor an overall system and provide coordination between network elements,the control plane, and the transport plane. As examples, the managementplane may include an element management system (EMS), which handles oneor more network elements from the perspective of the elements, a networkmanagement system (NMS), which handles many devices from the perspectiveof the network, and an operational support system (OSS), which handlesnetwork-wide operations.

Modifications, additions or omissions may be made to optical transportnetwork 101 without departing from the scope of the disclosure. Forexample, optical transport network 101 may include more or fewerelements than those depicted in FIG. 1. Also, as mentioned above,although depicted as a point-to-point network, optical transport network101 may comprise any suitable network topology for transmitting opticalsignals such as a ring, a mesh, or a hierarchical network topology.

As discussed above, the amount of information that may be transmittedover an optical network may vary with the number of optical channelscoded with information and multiplexed into one signal. Accordingly, anoptical fiber employing a WDM signal may carry more information than anoptical fiber that carries information over a single channel. Besidesthe number of channels and number of polarization components carried,another factor that affects how much information can be transmitted overan optical network may be the bit rate of transmission. The higher thebit rate, the greater the transmitted information capacity. Achievinghigher bit rates may be limited by the availability of wide bandwidthelectrical driver technology, digital signal processor technology andincrease in the required OSNR for transmission over optical transportnetwork 101.

As shown in FIG. 1, optical transport network 101 may employ a digitalwrapper technology to encapsulate existing frames of data, which mayoriginate in a variety of native protocols, and may add packetizedoverhead for addressing, management, and quality assurance purposes. Theresulting optical signal, in the form of optical data units (ODUs) maythen be transported using individual optical wavelengths by opticaltransport network 101. The packetized overhead may be used to monitorand control the optical signals being transported using any of a varietyof different protocols. In particular embodiments, operation of opticaltransport network 101 is performed according to optical transportnetworking (OTN) standards or recommendations promulgated by theInternational Telecommunications Union (ITU), such as ITU-TG.709—“Interfaces for the Optical Transport Network” and ITU-TG.872—“Architecture of the Optical Transport Network”, among others. Theoptical wavelengths in OTN may rely on a hierarchical implementation oftime-division multiplexing (TDM) to optimize carrier wavelengthefficiency.

As a result of the hierarchical TDM arrangement of the optical signalsin OTN, OTN switching may be performed at different sub-wavelength bitrates along optical transport network 101. As used herein, OTN switchingrefers to switching ODU paths of different bit rates with the ODU beingthe atomic unit of switching. In contrast, Internet protocol (IP)switching, such as by an IP router, refers to switching of networksignals where an individual IP packet is the atomic unit of switching.In OTN switching, such as in optical transport network 101, an ODUremains in the optical domain outside of an OTN switch from networkingress to network egress. Within the OTN switch, an ODU may be accessedas an electrical domain object and OTN switching may include electricalswitching technology.

It is noted that while OTN switching does generally take place in theDWDM domain, ROADMs and DWDM may be formally referred to as layer0technologies (in The Basic Reference Model for Open SystemsInterconnection, also referred to as the OSI Reference Model). Incontrast, OTN may be described as a layer1 technology in the OSIReference Model, which may operate independently of the opticalwavelength domain (DWDM). For example, an OTN switch may theoreticallyoperate over dark fiber, galvanic conductors (such as copper), or over awireless medium (such as a millimeter-scale wave, or radio frequencies).

In general, the term “distributed” may refer to multiple nodes, ornetwork elements (NEs), interconnected by a network and a set ofcollaborating nodes (or NEs). As used herein, the term “disaggregated”may refer to a NE in a distributed network that is further reorganizedinto a set of disaggregated sub-components in a physical sense, ascompared to an aggregated physical structure, while maintaining thefunctionality of an integrated NE in a logical sense. In someembodiments, the disaggregated sub-components may be made openlyaccessible, in contrast to the aggregated physical structure.

In contrast to the centralized and embedded nature of an OTN switch,which is a unitary device at a single central location, a disaggregatedOTN switching system is disclosed herein. The disaggregated OTNswitching system disclosed herein may enable disaggregation of the coreswitching functionality with the network interface functionality. Thedisaggregated OTN switching system disclosed herein may enable OTNswitching by relying on an internal Ethernet switching core (alsoreferred to herein as an “Ethernet fabric”). The disaggregated OTNswitching system disclosed herein may accordingly enable rapidcustomized configuration of a particular switching functionality at aparticular location or at different remote locations. The disaggregatedOTN switching system disclosed herein may enable much lower cost OTNswitching than by using an OTN switch. The disaggregated OTN switchingsystem disclosed herein may enable a much greater scalability ascompared to the fixed switching capacity that is inherent in an OTNswitch, because the Ethernet fabric employed may be external networkinfrastructure, such as data center switching systems, that can beexpanded to a desired capacity. The disaggregated OTN switching systemdisclosed herein may be implemented using a plurality of plug-inuniversal (PIU) modules that provide interfacing and transceivingfunctionality between various OTN signals and Ethernet signals. Thedisaggregated OTN switching system disclosed herein may be furtherimplemented using PIU blade chassis that have interface slots populatedby a number of PIU modules, which are interconnected, powered, andcontrolled using the PIU blade chassis. Certain ones of PIU modulesdisclosed herein may enable localized direct OTN switching functionalityby interconnecting two or more PIU modules in a loop-back configuration,without the use of a core Ethernet fabric.

Referring now to FIG. 2, a block diagram of selected elements of anembodiment of a disaggregated OTN switching system 200 is illustrated.Disaggregated OTN switching system 200 in FIG. 2 may be implemented forexternal switching of optical signals associated with optical transportnetwork 101 (see FIG. 1) and is a schematic diagram for descriptivepurposes and is not drawn to scale or perspective. External switching ofoptical signals refers to switching ODU paths of different bit rateswith an ODU being the atomic unit of switching, where the different bitrates may be sub-wavelength bit rates, and the ODU remains in theoptical domain outside of an OTN switch 230 from network ingress tonetwork egress. It is noted that within disaggregated OTN switchingsystem 200, an ODU may be accessed as an electrical domain object andOTN switching may include electrical switching technology.

As shown in FIG. 2, disaggregated OTN switching system 200 is configuredto function as OTN switch 230, in which optical signals having opticaldata unit (ODU) stream headers 232 connected to PIU modules 204 may beinterconnected and logically switched among PIU modules 204. At the coreof disaggregated OTN switching system 200 is an Ethernet fabric 220.Each of PIU modules 204 may function as a transceiver, with OTN inputsand outputs 210 (shown as cylindrical ports) being respectivelyconverted from ODUs each having an ODU header 224 to Ethernet packetseach having an Ethernet switching header 222 that are then switchable byone or more Ethernet switches 212. Ethernet fabric 220 may employEthernet switches 212 in any kind of Ethernet switching architecture orEthernet switching domain. In various embodiments, Ethernet fabric 220may be implemented as a hierarchical spine-leaf architecture, which hasbecome commonplace in many data center rack domains. Thus, each rack mayhave a so-called top-of-rack (TOR) leaf switch that operates at arelative low data throughput capacity, while the TOR leaf switches arethen interconnected using a spine switch that operates at a relativelyhigh data throughput capacity. In this manner, Ethernet fabric 220 maybe hierarchically implemented using different numbers of TOR leafswitches and spine switches for any given network switching application,including aggregation into very large throughput Ethernet fabrics 220that may have data throughput capacity of several dozens of terabytes,or even greater.

The interconnections between PIU modules 204 and Ethernet fabric 220 maybe copper cabled connections, such as 1000BASE-CX, 1000BASE-KX,1000BASE-T, and 1000BASE-TX for 1 GB Ethernet; such as 10GBASE-CX4,small form factor pluggable+ (SFP+), 10GBASE-T, and 10GBASE-KX4 for 10GB Ethernet; and such as 100GBASE-CR10, 100GBASE-CR4, 100GBASE-KR4, and100GBASE-KP4 for 100 GB Ethernet, among other potential types ofcopper-cable based ports. In some embodiments, the interconnectionsbetween PIU modules 204 and Ethernet fabric 220 may be optical fiberEthernet connections that are supported according to a variety ofEthernet standards for optical Ethernet ports. For example, for 100 GBEthernet interconnections to Ethernet fabric, the interconnections maybe any one or more of 100GBASE-SR10, 100GBASE-SR4, 100GBASE-LR4,100GBASE-ER4, 100GBASE-CWDM4, 100GBASE-PSM4, 100GBASE-ZR, 100GBASE-KR4,and 100GBASE-KP4. For example, for up to 400 GB Ethernetinterconnections to Ethernet fabric 220, the interconnections may be anyone or more of 400GBASE-SR16, 400GBASE-DR4, 400GBASE-FR8, and400GBASE-LR8. Furthermore, in certain embodiments, interconnections toEthernet fabric 220 may utilize FlexEthernet (FlexE) in order to mixdifferent transmission rates across Ethernet fabric 220.

Among the form factors for ports used in PIU modules 204 are quad smallform-factor pluggable (QFSP), C form-factor pluggable (CFP, CFP2), andSFP+. For example, on the OTN line side, CFP2 ports supporting analogcoherent optics (ACO) may be used in PIU modules 204, such as for 100gigabit (100 G) or 200 gigabit (200 G) coherent OTN connections.

Each PIU module 204 in disaggregated OTN switching system 200 is furtherequipped with an OTN over Ethernet (OTNoE) module 206, respectively,which may be an application specific integrated circuit (ASIC), an ASSP(application specific standard product), or a field-programmable gatearray (FPGA) that is customized for a particular purpose. OTNoE module206 in PIU module 204 may provide specific functionality to enableoverall operation of disaggregated OTN switching system 200 as OTNswitch 230. OTNoE module 206 may be enabled to implement, in the contextof disaggregated OTN switching system 200, various types of OTNfunctionality over Ethernet fabric 220. OTNoE module 206 may support orenable functionality for OTN path redundancy and path protectionswitching using Ethernet fabric 220. OTNoE module 206 may support orenable functionality for concatenation of OTN path protection domains.OTNoE module 206 may support or enable functionality for distribution ofOTN network paths and ODUs associated with the network paths over a 1:NEthernet fabric connections, where 1 Ethernet switch 212 is used toprotect N other working Ethernet switches 212 in case any one of the Nworking Ethernet switches 212 has a failure or indicates performance ofa maintenance operation that may result in an offline state.Furthermore, both 1:N and 0:N protection schemes may be supported. Giventhe nature of very high speed switching for both OTN applications andfor Ethernet fabrics 220, as well as the cost and complexity of usingexternal memory with OTNoE module 206, a latency delay variation may beexperienced among Ethernet switches 212. The latency delay variation (orjitter) by Ethernet fabric 220 may be an important factor to considerwhen choosing an ODU path distribution scheme and a particular Ethernetfabric 220 when a protection scheme is used. OTNoE module 206 maysupport or enable functionality for ensuring ODU path and data integrityover Ethernet fabric 220, even when jitter occurs over Ethernet fabric220. OTNoE module 206 may support or enable functionality for switchinghigher level ODUs over Ethernet fabric 220, even when the datathroughput for the higher level ODUs is larger than the underlyingEthernet ports in Ethernet fabric 220. The OTNoE module 206 may supportor enable functionality for compressing OTN traffic to provide moreefficient connections to Ethernet fabric 220 while compensating forjitter and bit error rate (BER) losses that may occur over Ethernetfabric 220, in order to enable using Ethernet fabric 220 for OTNswitching.

Finally, in FIG. 2, an OTN switch network element controller 214 (seealso FIGS. 3 and 4) is shown that coordinates operation of PIU bladechassis 202, PIU modules 204, and Ethernet fabric 220. OTN switchnetwork element controller 214 may be a software-defined networking(SDN) controller, a micro-controller unit (MCU), a virtualmicro-controller unit (vMCU), or various types of controllers.Specifically, functionality in the OTN switch network element controller214 may be used to communicate with PIU chassis 202 and Ethernet fabric220 for OTN switching operations. The OTN switch network elementcontroller 214 may accordingly configure switching paths and switchingconfigurations, using software commands, to enable operation ofdisaggregated OTN switching system 200 as an OTN switch 230.

Referring now to FIG. 3, a block diagram of selected elements of anembodiment of OTN switch network element controller 300 is illustrated.OTN switch network element controller 300 in FIG. 3 may be implementedto control disaggregated OTN switching system 200 (see FIG. 2) and is aschematic diagram for descriptive purposes.

In FIG. 3, OTN switch network element controller 300 is represented as acomputer system including physical and logical components forimplementing disaggregated OTN switching system 200, as describedherein, and may accordingly include processor 301, memory 310, andnetwork interface 322. Processor 301 may represent one or moreindividual processing units and may execute program instructions,interpret data, process data stored by memory 310 or OTN Switch NetworkElement Controller 300. It is noted that OTN switch network elementcontroller 300 may be implemented in different embodiments. For example,in some embodiments, OTN switch network element controller 300 may beimplemented using a network element. In particular embodiments, memory310 may represent a software controller 320 executing on processor 301.

In FIG. 3, memory 310 may be communicatively coupled to processor 601and may comprise a system, device, or apparatus suitable to retainprogram instructions or data for a period of time (e.g.,computer-readable media). Memory 310 may include various types ofcomponents and devices, such as random access memory (RAM), electricallyerasable programmable read-only memory (EEPROM), a PCMCIA card, flashmemory, solid state disks, hard disk drives, magnetic tape libraries,optical disk drives, magneto-optical disk drives, compact disk drives,compact disk arrays, disk array controllers, or any suitable selectionor array of volatile or non-volatile memory. Non-volatile memory refersto a memory that retains data after power is turned off. It is notedthat memory 310 may include different numbers of physical storagedevices, in various embodiments. As shown in FIG. 3, memory 310 mayinclude software controller 320, among other applications or programsavailable for execution.

Referring now to FIG. 4, a block diagram of selected elements of anembodiment of OTN switch network element controller 400 is illustrated.FIG. 4 shows further details of software controller 320 for performingSDN operations related to disaggregated OTN switching system 200, asdescribed above.

In FIG. 4, software controller 320 is shown including a repository thatmay store any of various different abstraction models 412, selected asexamples among other abstraction models for descriptive clarity. In someembodiments, abstractions models 412 are defined using YANG, which is adata modeling language for modeling configuration and state data used tomanage network devices through a network configuration protocol(NETCONF). Specifically, abstraction model 412 may include a serviceabstraction model that may model configuration and state data fornetwork services used with optical transport network 101. Abstractionmodel 412 may include a network abstraction model that may modelconfiguration and state data for network connections used with opticaltransport network 101. Abstraction model 412 may include a deviceabstraction model that may model configuration and state data fornetwork devices 420 used in optical transport network 101. Controlfunctions 410 may represent various control functions for networkservices, network connections, and network devices 420. API 408 mayenable control logic 406 to access control functions 410 for networkservices, network connections, and network devices 420.

As shown in OTN switch network element controller 400, API 414 mayenable communication between control logic 406, as well as externalapplications 416. Some non-limiting examples of external applications416 that may be used with software controller 320 include orchestrators(NCX, Anuta Networks, Inc., Milpitas, Calif., USA; Exanova ServiceIntelligence, CENX, Ottawa, Canada), workflow managers (SalesforceService Cloud, salesforce.com, Inc., San Franciso, Calif., USA;TrackVia, TrackVia, Inc., Denver, Colo., USA; Integrify, Integrify Inc.,Chicago, Ill., USA); and analytics applications (Cloud Analytics Engine,Juniper Networks, Inc., Sunnyvale, Calif., USA; Nuage NetworksVirtualized Services Directory (VSD), Nokia Solutions and Networks Oy,Espoo, Finland).

In implementations of OTN switch network element controller 400, controllogic 406 may comprise internal control logic that remains proprietary,internal, or administratively protected within software controller 320.Non-limiting examples of internal or protected portions of control logic406 may include complex proprietary algorithms, such as for pathcomputation, and private business logic, such as billing algorithms ofthe network operator. In disaggregated OTN switching system 200, controllogic 406 may include functionality for communicating with PIU chassis202 and Ethernet fabric 220, as described above.

Furthermore, software controller 320 may interact with various networkdevices 420 using different network protocols. For example, softwarecontroller 320 may interact with network device 420 using softwareprotocol 422 that is a NETCONF protocol, a command line interface (CLI),or a simple network management protocol (SNMP). Network devices 420 mayrepresents routers, switches, or network elements that are included inoptical transport network 101. As noted above, network abstractionmodels 412 may be repositories, such as databases with representationsof functions supported by software controller 320, while the actualimplementation of the functions is performed by control functions 410.Accordingly, control functions 410 may utilize the different softwareprotocols 422 to access network devices 420.

It is noted that network devices 420 and software protocols 422 areshown in a logical view in FIG. 4 not a physical view. The actualphysical connections between network devices 420 and software controller320 may be different in different embodiments, such as using one or morenetwork connections.

Referring now to FIG. 5, a representation of selected elements of anembodiment of a PIU chassis 500 is illustrated. PIU chassis 500 may be arack-mounted enclosure having an internal bus and an internal processor.PIU chassis 500 may receive PIU modules 204 via individual slots thatconnect PIU module 204 to the internal bus. The internal bus may providepower and coordination among PIU modules 204. In certain embodiments,PIU chassis 500 includes a network connection for direct communicationto the OTN switch network element controller 214. As shown PIU chassis500 has four slots that may be populated with individual PIU modules204. It is noted that in different embodiments, PIU chassis 500 may beimplemented with different numbers of slots and may be implemented indifferent form factors. It is noted that PIU modules 204 may have frontside network connections for access while PIU modules 204 populates aslot in PIU chassis 500.

Referring now to FIG. 6A, a block diagram of selected elements of anembodiment of a coherent PIU module 601 is illustrated. FIG. 6A is aschematic illustration. Coherent PIU module 601 may populate one slotPIU chassis 500. In the exemplary embodiment shown in FIG. 6A, coherentPIU module 601 is implemented with OTN inputs and outputs 210, twoanalog coherent optical (ACO) transceivers 604, for example, thatsupport 100 G or 200 G OTN lines and 100 G Ethernet, and a 16×10 GEthernet/4×40 G Ethernet port 208, and 100 G Ethernet port 208. CoherentPIU module 601 may further include a DSP 606 and an OTN framer+switch608, along with OTNoE module 206 on the 100 G Ethernet side, asdescribed above. Coherent PIU module 601 may include various connectorports for optical or copper wire based connections, as described above.

Referring now to FIG. 6B, a block diagram of selected elements of anembodiment of a client PIU module 602 is illustrated. FIG. 6B is aschematic illustration. Client PIU module 602 may populate one slot inPIU chassis 500. In the exemplary embodiment shown in FIG. 6B, clientPIU module 602 is implemented with OTN inputs and outputs 210, a 16×10 GEthernet/4×40 G Ethernet port 208, and 100 G Ethernet port 208. ClientPIU module 602 may further include OTN framer+switch 608, along withOTNoE module 206 on the 100 G Ethernet side, as described above. ClientPIU module 602 may include various connector ports for optical or copperwire based connections, as described above.

Referring now to FIG. 6C, a block diagram of selected elements of anembodiment of a high density PIU module 603 is illustrated. FIG. 6C is aschematic illustration. High density PIU module 603 may populate twoslots in PIU chassis 500. In the exemplary embodiment shown in FIG. 6C,high density PIU module 603 is implemented with two submodules that maybe similar to coherent PIU module 601, but where each sub-module maysupport 2×100 G OTN lines having OTN inputs and outputs 210. Highdensity PIU module 603 may further include two analog coherent optical(ACO) transceivers 604, 1×40 G Ethernet/10×10 G Ethernet client ports208, 16×10 G Ethernet ports 208, two DSPs 606 and two OTNframer+switches 608, along with two OTNoE modules 206 on the 100 GEthernet side, as described above. High density PIU module 603 mayinclude various connector ports for breaking out various optical orcopper wire based connections, as described above.

Referring now to FIG. 7A, a block diagram of selected elements of anembodiment of local switching configurations using two directlyinterconnected PIU modules 604 without a core Ethernet fabric 220 areshown. In the configurations shown in FIG. 7A, the OTN framer+switch 608may perform OTN switching, along with OTNoE modules 206 among theconnected modules. Although certain direct connections are shown in FIG.7A, it will be understood that local switching configurations using PIUmodules 204 may utilize internal connections as well as mesh connectionconfigurations, in which 3 or 4 PIU modules 204 are directlyinterconnected to enable cross-connections for all participants in themesh. For example, FIG. 7B illustrates a block diagram of selectedelements of an embodiment of local switching using four directlyinterconnected PIU modules 204 without a core Ethernet fabric 220. Inthis manner, certain local OTN bi-directional switching functionalityusing multiple switching nodes may be realized with low complexity andcost.

Referring now to FIG. 8, a block diagram of an example of optical dataunit (ODU) forwarding over an Ethernet fabric in an OTN switching system800 is illustrated. In FIG. 8, OTN switching system 800 is shown in aschematic representation and is not drawn to scale or perspective. It isnoted that, in different embodiments, OTN switching system 800 may beoperated with additional or fewer elements.

In FIG. 8, optical data units (ODU) including ODU 834 may enter OTNswitching system 800 in sequence (834-1, 834-2, 834-3, 834-4) at ingressPIU module 204-1, depicted by ODU switched connection 836-1 representingthe arrival of in sequence ODUs 834 at ingress PIU module 204-1. ODU 834s may exit OTN switching system 800 at egress PIU module 204-2 in thesame sequence as in ODU switched connection 836-1. In other words, ODUswitched connection 836-1 maintains the same sequence of departure ofODU 834 s at egress PIU module 204-2 as their in sequence arrival atingress PIU module 204-1.

In FIG. 8, OTN switching system 800 may establish ODU switchedconnection 836-1 to enable ODU forwarding of Ethernet packets overEthernet fabric 220-1 from PIU module 204-1 to PIU module 204-2. ODUswitched connection 836-1 may include connections from each of ports 208of ingress PIU module 204-1 to each of Ethernet switches 212 includingthe connection from port P1 208-1 to Ethernet switch 212-1, theconnection from port P2 208-2 to Ethernet switch 212-2, the connectionfrom port P1 208-3 to Ethernet switch 212-3, and the connection fromport P1 208-4 to Ethernet switch 212-4. ODU switched connection 836-1may also include connections from each of Ethernet switches 212 to eachof ports 208 of egress PIU module 204-2 including the connection fromEthernet switch 212-1 to port 208-1, the connection from Ethernet switch212-2 to port 208-2, the connection from Ethernet switch 212-3 to port208-3, and the connection from Ethernet switch 212-4 to port 208-4. Itis noted that in different embodiments, OTN switching system 800 mayestablish multiple ODU switched connections 836 (not shown in FIG. 8),each ODU switched connection 836 to enable ODU forwarding over Ethernetfabric 220-1 from one PIU module 204 of multiple PIU modules 204 toanother PIU module 204 of multiple PIU modules 204.

OTNoE 206-1 of OTN switching system 800 may receive in sequence ODUs 834at ingress PIU module 204-1. Each ODU 834 may include ODU header 224having information that indicates an ingress (also referred herein as asource) PIU module 204 and an egress (also referred herein as adestination) PIU module 204. OTNoE 206-1 uses the information associatedwith each ODU 834 to determine the destination egress PIU module 204. Inthe example embodiment, ODUs 834 each include information that indicatesingress PIU module 204 is PIU module 204-1 and egress PIU module 204 isPIU module 204-2. It is noted that in different embodiments, ODU headers224 of associated ODUs 834 each may include information that indicatesthe associated ingress PIU module 204 is the same or different amongstODUs 834 and the associated egress PIU module 204 is the same ordifferent amongst ODUs 834.

In OTN switching system 800, each PIU module 204 is assigned its ownunique identifier. The unique identifier may be assigned by OTN switchnetwork element controller 214 during a configuration process of OTNswitching system 800 or by OTN switch network element controller 214when each PIU module 204 is added to OTN switching system 800. PIUmodule identifier may be a media access control (MAC) address, a virtuallocal area network (VLAN) identifier, and the like. In the exampleembodiment, PIU module 204-1 is assigned MAC address M1 826-1 and PIUmodule 204-2 is assigned MAC address M2 826-2.

OTNoE 206-1 determines from information included in each ODU header 224of associated ODUs 834 that the destination egress PIU module 204 is PIUmodule 204-2 and generates each Ethernet packet 828 (PKT) including PKT828-1 through PKT 828-4 from each corresponding ODU 834-1 through ODU834-4, respectively. In the example embodiment, there is a one to onecorrespondence between ODU 834-1 through ODU 834-4 and PKT 828-1 throughPKT 828-4. Each generated PKT 828 includes an Ethernet switching header222 which may include information from each ODU header 224 of associatedODUs 834. Each Ethernet switching header 222 of generated PKTs 828 mayalso include information that indicates the source MAC address of theingress PIU module and the destination MAC address of the egress PIUmodule, where the source MAC address is MAC address M1 826-1 of ingressPIU module 204-1 and the destination MAC address is MAC address M2 826-2of egress PIU module 204-2, as indicated by M1 and M2 of PKTs 828. Thesource and destination MAC addresses may be a unicast MAC address, amulticast MAC address, a broadcast MAC address, and the like. Thegenerated PKTs 828 may further include a sequence number assigned toeach PKT 828 that indicates the in sequence order of PKTs 828 thatcorresponds to the in sequence arrival order of ODUs 834. The sequencenumber of each packets is utilized by the destination egress PIU module204 to recover and maintain the in sequence arrival order of ODUs 834 atPIU module 204-1, described in further detail below. The generated PKTs828 may be for transmission via ODU switched connection 836-1corresponding to ingress PIU module 204-1 and egress PIU module 204-2.

OTNoE 206-1 selects one of ports 208 for transmission of each PKT 828 ofPKTs 828 and transmits each PKT 828 of PKTs 828 from its selected port208 of PIU module 204-1 over Ethernet switch 212 corresponding to theselected port 208. In the example embodiment, OTNoE 206-1 selects portP1 208-1 for transmission of PKT 828-4 and transmits PKT 828-4 from portP1 208-1 over Ethernet switch 212-1, depicted by the dashed arrow fromport P1 208-1 to Ethernet switch 212-1. Similarly, OTNoE 206-1 selectsport P2 208-2 and transmits PKT 828-1 from port P2 208-2 over Ethernetswitch 212-2, depicted by the dashed arrow from port P1 208-2 toEthernet switch 212-2, selects port P3 208-3 and transmits PKT 828-3from port P3 208-3 over Ethernet switch 212-3, depicted by the dashedarrow from port P3 208-3 to Ethernet switch 212-3, and selects port P4208-4 and transmits PKT 828-2 from port P4 208-4 over Ethernet switch212-4, depicted by the dashed arrow from port P4 208-4 to Ethernetswitch 212-4. The connections between ports P1 208-1 through ports P4208-4 and Ethernet switches 212-1 through 212-4 allow an ingress PIUmodule 204 to transmit PKTs 828 in parallel on all available Ethernetswitches 212. When all N Ethernet switches 212 are available duringnormal operation, Ethernet fabric 220-1 is in a 0:N load sharing mode.When one of Ethernet switches 212 is unavailable, e.g. due to anequipment failure, an interconnect cable failure, or maintenance, aningress PIU module 204 transmits PKTs 828 on all remaining availableEthernet switches 212, and therefore, realize fabric protection Ethernetswitching.

OTNoE 206-2 may include a re-sequencing buffer 870-1 to store PKTs 828received at ports 208 of PIU module 204-2. OTNoE 206-2 receives PKTs 828from Ethernet switches 212 at ports 208 of PIU module 204-2corresponding to ports P1 208 of PIU module 204-1 and stores PKTs 828 atre-sequencing buffer 870-1 of OTNoE 206-2. In the example embodiment,OTNoE 206-2 receives PKT 828-4 at port P1 208-5, PKT 828-1 at port P2208-6, PKT 828-3 at port P3 208-7, and PKT 828-2 at port P4 208-8 andstores PKT 828-1 through PKT 828-4 at re-sequencing buffer 870-1. Duringoperation, Ethernet fabric 220-1 may be in load sharing mode, wheremultiple PKTs 828 may be in transmission over multiple Ethernet switches212 resulting in arrival packet jitter, which may be intrinsic packetjitter or extrinsic packet jitter.

Intrinsic packet jitter may be due to differences amongst PIU modules204, interconnects, e.g. cables, Ethernet switches 212, and othercomponents that may comprise OTN switching system 800. Extrinsic packetjitter may be due to multiple ingress PIU modules 204 transmittingmultiple Ethernet packets 828 to the same port of the same egress PIUmodule 204 resulting in varied Ethernet packet arrival times. In otherwords, intrinsic packet jitter may be defined as originating from allcauses other than Ethernet packet 828 collisions or retransmissions,which may be defined as causes for extrinsic packet jitter. Inparticular, OTN switching system 800 is designed and operated tominimize or eliminate extrinsic packet jitter, such that variations inegress receive time 838 may be assumed to be relatively small andoriginate from intrinsic packet jitter.

Ethernet fabric 220-1 operating in load sharing mode may result inEthernet packets 828 arriving at ports 208 of PIU module 204-2 out ofsequence to their transmission sequence from PIU module 204-1. In theexample embodiment, PKT 828-1 arrives first as depicted by its arrivaltime with respect to egress receive time 838, PKT 828-3 arrives next,PKT 828-2 arrives next, and PKT 828-4 arrives last. As illustrated, PKTs828 also overlap each other with respect to egress receive time 838.

OTNoE 206-2 re-assembles ODU 834-1 through ODU 834-4 includingre-assembling each ODU header 224 of each ODU 834 from PKT 828-1 throughPKT 828-4 stored at re-sequencing buffer 870-1. OTNoE 206-2 re-sequencesODU 834-1 through ODU 834-4 into the same sequence that corresponds tothe in sequence arrival order of ODUs 834 at PIU module 204-1 based onthe sequence number assigned to each PKT 828 that corresponds to the insequence arrival order of ODUs 834. OTNoE 206-2 re-assembles each ODUheader 224 of each ODU 834 based on information included in eachEthernet switching header 222 of each PKT 828. Once the ODUs 834 arere-assembled and re-sequenced, the ODUs 834 may exit OTN switchingsystem 800 at egress PIU module 204-2 in the same sequence as theyentered OTN switching system 800 at ingress PIU module 204-1.

Referring now to FIG. 9, a block diagram of an example of ODU 834forwarding using a cyclical walk sequence over an embodiment of Ethernetfabric 220-1 in an OTN switching system 900 is illustrated. In FIG. 9,OTN switching system 900 is shown in a schematic representation and isnot drawn to scale or perspective. It is noted that, in differentembodiments, OTN switching system 900 may be operated with additional orfewer elements.

In one or more embodiments, OTN switching system 900 may establish oneor more ODU switched connections 836 to enable ODU forwarding ofEthernet packets 828 over Ethernet fabric 220-1 from one or more ingressPIU modules 204 to one or more egress PIU modules 204, as previouslydescribed. In the example embodiment, OTN switching system 900 mayestablish a first ODU switched connection 836-1 from ingress PIU module204-1 to egress PIU module 204-2 and a second ODU switched connection836-2 from ingress PIU module 204-3 to egress PIU module 204-2. Thefirst ODU switched connection 836-1 includes connections from each ofports P1 208-1 through P4 208-4 of ingress PIU module 204-1 to each ofthe corresponding Ethernet switches 212-1 through 212-4, and connectionsfrom each of Ethernet switches 212-1 through 212-4 to each of thecorresponding ports P1 208-5 through P4 208-8 of egress PIU module204-2. Similarly, the second ODU switched connection 836-2 includesconnections from each of ports P1 208-9 through P4 208-8 of ingress PIUmodule 204-3 to each of the corresponding Ethernet switches 212-1through 212-4, and connections from each of Ethernet switches 212-1through 212-4 to each of the corresponding ports P1 208-5 through P4208-8 of egress PIU module 204-2.

In one or more embodiments, ingress PIU module 204 having ODU switchedconnection 836 may utilize a method to select an Ethernet switch 212 ofa number M of Ethernet switches 212 (also referred herein as an Ethernetplane) for Ethernet packet transmissions based on cyclical walk sequence944, where cyclical walk sequence 944 is associated with ODU switchedconnection 836. Cyclical walk sequence 944 may include a sequentialorder of a number M unique port identifiers, where each of the M portidentifiers are respectively associated with each of M ports 208 ofingress PIU module 212. The M unique port identifiers may be in one of anumber N different possible sequential orders in the cyclical walksequence 944, where the N different sequential orders is equal to Mfactorial divided by M (M!/M). In an embodiment of the method, ingressPIU module 204 may select a first port 208 through an M^(th) port 208 ofingress PIU module 204 and their corresponding first through M^(th)Ethernet switches 212 for transmission of first through M^(th) insequence Ethernet packets 828 based on first through M^(th) in sequenceport identifiers of cyclical walk sequence 944. Ingress PIU module 204may similarly select a next port 208 and its next Ethernet switch 212for transmission of the M+1^(th) in sequence Ethernet packet 828 basedon the first in sequence port identifier of cyclical walk sequence 944,where the method wraps from the M^(th)/last in sequence port identifierback to the first in sequence port identifier of cyclical walk sequence944. In another embodiment of the method, the port selection may bebased on a random port identifier of cyclical walk sequence 944.

In one or more embodiments, an ingress PIU module 204 is assigned acyclical walk sequence 944 for each OTN switched connection 836 ofingress PIU module 204. In an embodiment, an egress PIU module 204selects a cyclical walk sequence 944 and sends it to ingress PIU module204, where it is received and stored at ingress PIU module 204. Inanother embodiment, OTN switch network element controller 214, on behalfof egress PIU module 204, selects cyclical walk sequence 944 to bestored at egress PIU module 204. In FIG. 9, egress PIU module 204-2selects the first cyclical walk sequence 944-1 for PIU module 204-1 andthe second cyclical walk sequence 944-2 for PIU module 204-3.

When multiple ODU switched connections 836 of multiple ingress PIUmodules 204 each specify a single egress PIU module 204, each ingressPIU module 204 may utilize its assigned cyclical walk sequence 944 tohelp avoid each ingress PIU module 204 from synchronizing its selectionof the same Ethernet switch 212 with the other ingress PIU modules 204and inundating the single egress PIU module 204 with Ethernet packets828 from the same Ethernet switch 212. By assigning different cyclicalwalk sequences 944 to each ingress PIU module 204, the probability ofsynchronization of Ethernet switch selection may be minimized allowingOTN switching system 900 to use smaller buffers to endure a sizableinflux of Ethernet packets 828 when ingress PIU modules 204 dosynchronize their Ethernet switch selections.

In one or more embodiments, ingress PIU module 204 may maintain aconsecutive transmission port count for each port 208 of ingress PIUmodule 204. Each consecutive transmission port count is based onconsecutive Ethernet packet 828 transmissions to a single port 208,where Ethernet packet 828 transmissions are associated with differentEthernet switched connections 836. When ingress PIU module 204determines that it is transmitting an Ethernet packet 828 to the sameport 208 of ingress PIU module 204, the associated consecutivetransmission port count is incremented.

When the ingress PIU module 204 is selecting the next port 208 fortransmission of an in sequence Ethernet packet 828 based on the currentin sequence port identifier of cyclical walk sequence 944, and ingressPIU module 204 determines that the consecutive transmission port countexceeds a consecutive transmission port count threshold value associatedwith the next port 208, PIU module 204 skips the current in sequenceport identifier and selects the next port 208 based on the next insequence port identifier of cyclical walk sequence 944. Selecting ports208 in the above manner further minimizes the probability ofsynchronization of Ethernet switch 212 selection may be minimized

In FIG. 9, OTN switching system 900 may select, for ingress PIU module204-1, a first cyclical walk sequence 944-1 having a first sequentialorder of ports P1 208-1 through P4 208-4 for ODU forwarding of Ethernetpackets PKT 228-1 through PKT 228-4 from ingress PIU module 204-1 toegress PIU module 204-2, where the first cyclical walk sequence 944-1corresponds to the first ODU switched connection 836-1, ports P1 208-1through P4 208-4 of PIU module 204-1 correspond to Ethernet switches212-1 through 212-4, and each PKT 228-1 through PKT 228-4 correspond toODU 834-1 through ODU 834-4, respectively. Similarly, OTN switchingsystem 900 may select, for ingress PIU module 204-3, a second cyclicalwalk sequence 944-2 having a second sequential order of ports P1 208-9through P4 208-8 for ODU forwarding of Ethernet packets PKT 228-5through PKT 228-8 from ingress PIU module 204-3 to egress PIU module204-2, where the second cyclical walk sequence 944-2 corresponds to thesecond ODU switched connection 836-2, ports P1 208-9 through P4 208-8 ofPIU module 204-3 correspond to Ethernet switches 212-1 through 212-4,and each PKT 228-5 through PKT 228-8 correspond to ODU 834-5 through ODU834-8, respectively.

In FIG. 9, the first cyclical walk sequence 944-1 having four portidentifiers in the first sequential order of P1, P2, P3, and P4, whereport identifier P1 corresponds to P1 208-1 and Ethernet switch 212-1,port identifier P2 corresponds to P2 208-2 and Ethernet switch 212-2,port identifier P3 corresponds to P3 208-3 and Ethernet switch 212-3,and port identifier P4 corresponds to P4 208-4 and Ethernet switch212-4. The second cyclical walk sequence 944-2 having four portidentifiers in the second sequential order of P3, P1, P4, and P2, whereport identifier P1 corresponds to P1 208-9 and Ethernet switch 212-1,port identifier P2 corresponds to P2 208-10 and Ethernet switch 212-2,port identifier P3 corresponds to P3 208-11 and Ethernet switch 212-3,and port identifier P4 corresponds to P4 208-8 and Ethernet switch212-4. The first sequential order and the second sequential order aretwo of six possible sequential orders of four port identifiers, 4factorial divided by 4 (4!/4).

FIG. 9, OTNoE 206-1 of OTN switching system 900 may receive in sequenceODUs 834-1 through 834-4 at ingress PIU module 204-1 and OTNoE 206-2 mayreceive in sequence ODUs 834-5 through 834-8 at ingress PIU module204-3. ODUs 834-1 through 834-4 and 834-5 through 834-8 may haveconstant data bit rate, and may be well behaved because ODU switchedconnections 836 allow transmissions of Ethernet packets 828 from anumber M of ports 208 of an ingress PIU module 204 to M Ethernetswitches 212 at matching data rates, which may result in minimalcongestion that could cause delay variations to the recovery of ODUs 834at an egress PIU module 204. OTNoE 206-1 generates PKTs 828-1 through828-4 corresponding to ODUs 834-1 through 834-4, where ODUs 834-1through 834-4 are for transmission via the first ODU switched connection836-1. OTNoE 206-2 similarly generates PKTs 828-5 through 828-8corresponding to ODUs 834-5 through 834-8, where ODUs 834-5 through834-8 are for transmission via the second ODU switched connection 836-2.

OTNoE 206-1 may transmit the first PKT 828-1 from port P1 208-1 ofingress PIU module 204-1 to egress PIU module 204-2 via Ethernet switch212-1, where port P1 208-1 may be selected based on the first portidentifier, P1, in the first sequential order of the first cyclical walksequence 944-1. OTNoE 206-1 may transmit the second, third, and fourthPKTs 828-2 through 828-4 from ports P2 208-2, P3 208-3, and P4 208-4 ofingress PIU module 204-1 to egress PIU module 204-2 via Ethernetswitches 212-2, 212-3, and 212-4, where ports P2 208-2, P3 208-3, and P4208-4 may be selected based on the second, third, and fourth portidentifiers, P2, P3, and P4 in the first sequential order of the firstcyclical walk sequence 944-1. OTNoE 206-1 may transmit the nextin-sequence PKT 828 from port P1 208-1, where P1 208-1 may be selectedbased on wrapping around from the 4th/last in-sequence port identifierP4 to 1st/next port identifier P1 in the first sequential order of thefirst cyclical walk sequence 944-1.

Similarly, OTNoE 206-2 may transmit the first, second, third, and fourthPKTs 828-5 through 828-8 from ports P3 208-11, P1 208-9, P4 208-8, andP2 208-10 of ingress PIU module 204-3 to egress PIU module 204-2 viaEthernet switches 212-3, 212-1, 212-4, and 212-2 where ports P3 208-11,P1 208-9, P4 208-8, and P2 208-10 may be selected based on the first,second, third, and fourth port identifiers, P3, P1, P4, and P2 in thesecond sequential order of the second cyclical walk sequence 944-2.OTNoE 206-2 may transmit the next in-sequence PKT 828 from port P3208-11, where P3 208-11 may be selected based on wrapping around fromthe 4th/last in-sequence port identifier P2 to 1st/next port identifierP3 in the second sequential order of the second cyclical walk sequence944-2. In FIG. 9, the transmission of PKTs 828-1 through 828-4 from eachselected port 208 of ingress PIU module 204-1 to Ethernet switches 212and PKTs 828-5 through 828-8 from each selected port 208 of ingress PIUmodule 204-3 to Ethernet switches 212 are shown by dashed line arrowsbetween each selected port 208 and each corresponding Ethernet switch212.

In FIG. 9, OTNoE 206-3 of egress PIU module 204-2 may receive PKTs 828-1and 828-6 from ingress PIU modules 204-1 and 204-3 via Ethernet switch212-1 at port P1 208-5, depicted by the dashed arrow from Ethernetswitch 212-1 to port P1 208-5, PKTs 828-2 and 828-8 from Ethernet switch212-2 at port P2 208-6, depicted by the dashed arrow from Ethernetswitch 212-2 to port P1 208-6, PKTs 828-3 and 828-5 from Ethernet switch212-3 at port P3 208-7, depicted by the dashed arrow from Ethernetswitch 212-3 to port P1 208-7, and PKTs 828-4 and 828-7 from Ethernetswitch 212-4 at port P4 208-8, depicted by the dashed arrow fromEthernet switch 212-4 to port P1 208-8.

Referring now to FIG. 10, a block diagram of an example of multiple ODUswitched connections 836 through a single PIU module 204 in an OTNswitching system 1000 is illustrated. In FIG. 10, OTN switching system1000 is shown in a schematic representation and is not drawn to scale orperspective. It is noted that, in different embodiments, OTN switchingsystem 1000 may be operated with additional or fewer elements.

In OTN switching system 1000, multiple ODUs 834 may be bundled into asingle Ethernet packet 828 to minimize the latency of Ethernet packettransmissions over Ethernet fabric 220 and minimize the memory utilizedfor these transmissions. Only ODUs 834 associated with the same Ethernetswitched connection 836 may be bundled into a single Ethernet packet PKT828. The in sequence order of the ODUs 834 are bundled into the sameEthernet packet PKT 828 in in-sequence order.

OTNoE 206-1 and 206-2 may utilize ingress lookup table 1054 and egresslookup tables 1054, respectively, with local indexing to allow tablelookup functions to be deterministic within a single lookup cycle, asdescribed in further detail below. When OTNoEs 206 are implemented asfield programmable gate arrays (FPGA) and operated in this manner thesize of FPGA memory may be reduced. OTN switching system 1000 has asystem wide look up table of Ethernet switched connections 836 having arespective Ethernet switched connection 836 identification that includesentries for every Ethernet switched connection 836 for every ingress PIUmodule 204 and egress PIU module 204 in OTN switching system 1000. Toenable efficient table lookup functions to be performed by OTNoEs 206,each OTNoE 206 has a local ingress lookup table 1054 and an egresslookup table 1065 that are based on a portion of the system wide lookuptable, where the local ingress lookup table 1054 includes entries forthe Ethernet switched connections 836 associated with PIU module 204-1and its associated egress PIU module 204-2, and the local egress lookuptable 1065 includes entries for the Ethernet switched connections 836associated with egress PIU module 204-2 and its associated ingress PIUmodules 204-1. By distributing the system wide lookup table to a localingress lookup table 1054 for ingress OTNoE 206-1 and a local egresslookup table 1065 for egress OTNoE 206-2, the size of the local tablesare reduced and the lookup functions performed using these local lookuptables 1054 can be more efficient and deterministic.

OTNoE 206-1 of OTN switching system 1000 may receive a first ODU 834-1having a first ODU header 224-1 at an input control 1052, where thefirst ODU header 224-1 includes information that indicates PIU module204-2 is the destination egress PIU module 204-2 of the first ODU 834-1.Input control 1052 may perform a table lookup of ingress lookup table1054 (ILT) having a plurality of ILT entries to retrieve a first ILTentry based on egress PIU module 204-2 being the destination egress PIUmodule 204-2 of the first ODU 834-1. In an embodiment, input control1052 may perform the table lookup by using an ILT table index as adirect index to the first ILT entry of ILT 1054, where the ILT tableindex is based on egress PIU module 204-2. OTNoE 206-1 may store thefirst ODU 828 at a first location of a bundling buffer 1 1056-1, whereinformation included in the first ILT entry indicates that bundlingbuffer 1 1056-1 is to be used to store ODUs 834 associated with egressPIU module 206-2.

OTNoE 206-1 may receive a second ODU 834-2 that indicates egress PIUmodule 204-2 is the destination egress PIU module 204-2. Input control1052 may retrieve a second ILT entry of ILT 1054 based on ODU 834-2, asdescribed above. OTNoE 206-1 may store the second ODU 834-2 at a next insequence second location of bundling buffer 1 1056-1 based on the secondILT entry. Ethernet framer 1058 may generate a first Ethernet packet PKT828-1 including a first Ethernet switching header 222-1 corresponding tothe first ODU 834-1 and the second ODU 834-2 stored at the first andsecond locations of bundling buffer 1 1056-1 based on the first andsecond ILT entries, as previously described, where the first Ethernetswitching header 222-1 includes a first sequence number, the first ILTtable index, a first egress lookup table (ELT) index stored at the firstILT entry, and an Ethernet switched connection 836-1 stored at the firstILT entry. Ethernet framer 1058 may store the first Ethernet packet PKT828-1 at a first transmit queue of a plurality of transmit queuesincluding TX Q Ethernet Switch 1 1061-1 through TX Q Ethernet Switch 41061-1 for transmission to egress PIU module 206-2, where the firsttransmit queue TX Q Ethernet Switch 1 1061-1 is selected by OTNoE 206-1based on cyclical walk sequence 944-1.

Ethernet de-framer 1064 of egress OTNoE 206-2 may retrieve the firstEthernet packet PKT 828-1 stored at a first receive queue of a pluralityof receive queues including RX Q Ethernet Switch 1 1062-1 through RX QEthernet Switch 4 1061-4, where the first receive queue is RX Q EthernetSwitch 1 1062-1. OTNoE 206-2 may perform a table lookup of an ELT table1065 having a plurality of ELT entries to retrieve a first ELT entry ofELT 1065 based on the first ELT index of the first Ethernet switchingheader 222-1. OTNoE 206-2 may store the first Ethernet packet PKT 828-1at a first location of a re-sequencing buffer 870 of a plurality ofre-sequencing buffers 870 including re-sequencing buffer 1 870-2 throughre-sequencing buffer 3 870-4, where information included in the firstELT entry indicates that re-sequencing buffer 1 870-2 is to be used tostore Ethernet packets PKT 828 associated with the first ELT entry, andthe first location of re-sequencing buffer 1 870-2 is based on the firstsequence number of the first Ethernet switching header 222-1 of PKT828-1. Output control 1066 of OTNoE 206-2 may recover the first insequence ODU 834-1 and the second in sequence ODU 834-2 from Ethernetpacket PKT 828-1 based on information in Ethernet switching header 222-1of PKT 828-1 for transmission to the OTN networking system 1000.

OTN switching system 1000 may provide an environment to enable ODU clocksynchronization and clock recovery. In order to provide thisenvironment, OTN switching system 1000 may provide a single clock sourceto both ingress PIU module 204-1 and egress PIU module 204-1 so thatingress OTN framer+switch 608-1 and egress OTN framer+switch 608-2 mayuse the sigma-delta clock recovery method to recover an ODU clock usingtimestamps and a number of bytes transferred from ingress OTNframer+switch 608-1 to egress OTN framer+switch 608-2. Ingress PIUmodule 204-1 and egress PIU module 204-1 synchronize their ODU clockinterfaces to the single clock source provided by OTN switching system1000, or derived from OTN overhead information to enable ODU clockrecovery.

OTN switching system 1000 also provides in-sequence delivery of ODUs foreach ODU path, between ingress OTN framer+switch 608-1 and egress OTNframer+switch 608-2, even when Ethernet fabric 220 may produce out oforder delivery of Ethernet frames, as previously described.

OTN switching system 1000 further provides continuous ODU delivery toegress OTN framer+switch 608-2 in spite of any delays and jitters causedby Ethernet fabric 220, and buffers extra jitters that egress OTNframer+switch 608-2 may be unable to handle. There are two places thatde-jittering of ODUs occurs, a limited capability is provided in egressOTN framer+switch 608-2 and in a re-sequencing buffer 870 on a per ODUpath basis. Re-sequencing is performed on Ethernet packets 828 level.Once Ethernet packets 828 are re-assembled back in-sequence, Ethernetpacket 828 overhead can be removed and ODUs are delivered to egress OTNframer+switch 608-2. In order to prevent re-sequencing buffer 870underrun due to Ethernet fabric jitter, a watermark is set for eachre-sequencing buffer 870 to allow buffering of some Ethernet packets 828prior to delivery of ODUs to egress OTN framer+switch 608-2. Ethernetfabric jitter and Ethernet packet collision may also cause re-sequencingbuffer 870 to build up beyond the watermark. In order to reduce internalmemory requirements of OTNoE 206, re-sequencing buffers 870 are sharedfor all ODU paths of PIU modules 204, where re-sequencing buffers 870may have buffer descriptors that are queued, linked, to allow for moreefficient re-ordering. In OTN switching system 1000, per ODU pathback-pressure may occur between egress OTN framer+switch 608-2 andre-sequencing buffer 870. Egress OTNoE 206-2 delivers ODUs 834 to egressOTN framer+switch 608-2. Egress OTN framer+switch 608-2 may utilize flowcontrol to control the rate of emission of ODUs 834 for each specificODU path from egress OTNoE 206-2.

Ingress OTN framer+switch 608-1 provides the number of bytes of eachoriginal ODU in each subsequent Ethernet packet header between twoadjacent timestamps at egress OTN framer+switch 608-2, even in caseswhere there is ODU loss or ODU corruption. When an Ethernet packet 828is lost, the number of bytes in the next Ethernet packet header ofEthernet packet 828 on the same ODU path will be used to deliver thenumber of bytes missing on the ODU stream. The content of the lostEthernet packet 828 will be lost but the clock information can berecovered.

Referring now to FIG. 11, a block diagram of an embodiment of selectedelements of virtual slots in an Ethernet switch fabric 1100 isillustrated. It is noted that FIG. 11 is a schematic diagram fordescriptive purposes and is not drawn to scale or perspective. As shown,each Ethernet switch 212-5 through 212-8 includes 32 respective Ethernetswitch ports 1116 SP1-SP32. As shown in FIG. 11, each of the 32respective Ethernet switch ports 1116 SP1-SP32 may be divided into fourEthernet switch sub-ports 1120 for a total of 128 Ethernet switchsub-ports 1120 S1-S128. Each of Ethernet switch ports 1116 may include aquad small form-factor pluggable (QSFP) transceiver. For example, theQSFP transceiver of each Ethernet switch port 1116 may be a QSFP28transceiver. For instance, the QSFP28 transceiver of each Ethernetswitch port 1116 may support and enable a 100 GE (one hundred gigabitEthernet) connection to respective Ethernet switches.

In one or more embodiments, Ethernet switch sub-ports 1120 of respectiveEthernet switches 212 of Ethernet fabric 220-2 may form multiple virtualslots 1122. The multiple virtual slots 1122 may include a logicalaggregation of multiple consecutive Ethernet switch sub-ports 1120. Forexample, as illustrated, Ethernet switch sub-ports 1120 S1-S4 ofrespective Ethernet switches 212 may form virtual slots VS1 1122-1-VS41122-4. Other virtual slots 1122 may include other Ethernet switchsub-ports 1120, of respective Ethernet switches 212, to form othervirtual slots 1122. For example, Ethernet switch sub-ports 1120 S5-S8 ofrespective Ethernet switches 212 may form virtual slots VS5 1122-5-VS81122-8. Ethernet switch sub-port 1120 S125 may form virtual slot VS1251122-125. Ethernet fabric 220-2 may include a number P of Ethernetswitch sub-ports 1120 of Ethernet switches 212, where P is greater thanone. Ethernet fabric 220-2 may also include P virtual slots 1122corresponding to respective Ethernet switch sub-ports 1120 of Ethernetswitches 212. A variable k having a value ranging from 1 to P denotesthe kth Ethernet switch sub-port 1120 Sk corresponding to one of the PEthernet switch sub-ports 1120. The variable k also denotes the kthvirtual slot 1122 VSk having a virtual slot number of k corresponding toone of the P virtual slots.

A virtual slot 1122 may be associated with a virtual slot address thatis unique to the virtual slot 1122 in the Ethernet fabric 220-2, wherethe virtual slot address may be set to the Ethernet switch sub-portnumber k of the starting Ethernet switch sub-port 1120 of the virtualslot 1122. For example, virtual slot VS125 1122-125 starts with Ethernetswitch sub-port 1120 S125 having the Ethernet switch sub-port number 125and its virtual slot address is set to 125. In one or more otherembodiments, the virtual slot address may be set to the virtual slotnumber k of the starting virtual slot 1122 of the virtual slot. Forexample, the virtual slot address of virtual slots VS1 1122-1-VS4 1122-4may be set to 1 that is unique in Ethernet fabric 220-2. In anotherexample, virtual slots VS5 1122-5-VS8 1122-8 may be associated with avirtual slot address set to 5 that is also unique in Ethernet fabric220-2. In yet another example, virtual slot VS125 1122-125 may beassociated with a different virtual slot address, 5, which is alsounique in Ethernet fabric 220-2.

Referring now to FIG. 12, a block diagram of an example of ODU pathprotection in an embodiment of an OTN switching system 1200 isillustrated. In FIG. 12, OTN switching system 1200 is shown in aschematic representation and is not drawn to scale or perspective. It isnoted that, in different embodiments, OTN switching system 1200 may beoperated with additional or fewer elements.

In OTN switching system 1200, one or more ODU paths 1205 may beestablished for transmission of ODUs. One or more working ODU paths 1206may also be established for transmission of ODUs and one or moreprotection ODU paths 1207 may be established for sub-network connectionprotection (SNCP) which is a per path protection mechanism for workingODU paths 1206 using protection ODU paths 1207. The working ODU paths1206 may include a working ODU path 1206-1 that extends from a head endOTN switch 1201 to an OTN network 1204-2, and from OTN network 1204-2 toa tail end OTN switch 1202. Working ODU path 1206-1 may further beestablished from Ethernet fabric 220-3 to an egress PIU module 204-5,from egress PIU module 204-5 to an ingress PIU module 204-7 via OTNnetwork 1204-2, from ingress PIU module 204-7 to an Ethernet fabric202-4, and from Ethernet fabric 220-4 to an egress PIU module 204-9.Multiple protection ODU paths 1207 including a protection ODU path1207-1 may be established for SNCP. Protection ODU path 1207-1 may beestablished from Ethernet fabric 220-3 to an egress PIU module 204-6,from egress PIU module 204-6 to an ingress PIU module 204-8 via OTNnetwork 1204-2, from ingress PIU module 204-8 to Ethernet fabric 202-4,and from Ethernet fabric 220-4 to egress PIU module 204-9. ProtectionODU path 1207-1 protects working ODU path 1206-1 in case OTN switch 1202detects at least one of: a fault condition associated with working ODUpath 1206-1; an OTN command to perform a protection switch; a cessationof ODU transmission over working ODU path 1206-1; an impairment of ODUtransmission over working ODU path 1206-1, among other conditions.

In OTN switching system 1200, head end OTN switch 1201 may be connectedto ODU path 1205, working ODU path 1206-1, and protection ODU path1207-1. Tail end OTN switch 1202 may also be connected to ODU path 1205,working ODU path 1206-1, and protection ODU path 1207-1. Ingress PIUmodule 204-4 may be connected to ODU path 1205 and Ethernet fabric220-3, Egress PIU modules 204-5 and 204-6 may be connected to workingODU path 1206-1, protection ODU path 1207-1, and Ethernet fabric 220-3.Ingress PIU modules 204-7 and 204-8 may be connected to working ODU path1206-1, protection ODU path 1207-1, and Ethernet fabric 220-4, andEgress PIU module 204-9 may be connected to working ODU path 1206-1,protection ODU path 1207-1, Ethernet fabric 220-4, and ODU path 1205.Until a protection switch is performed, egress PIU module 204-5 andingress PIU module 204-7 may operate in a working state, and egress PIUmodule 204-6 and ingress PIU module 204-8 may operate in a protectionstate, described in further detail below.

During uni-directional SNCP switching operation of head end OTN switch1201, ingress PIU module 204-4 may receive in sequence ODUs 834 from OTNnetwork 1204-1 via ODU path 1205. Ingress PIU module 204-4 may utilize amulticast media access control (MAC) address to transmit Ethernetpackets 828 corresponding to the in sequence ODUs 834 to both egress PIUmodule 204-5 and egress PIU module 204-6 via Ethernet fabric 220-3. Thetransmission of Ethernet packets 828 from ingress PIU module 204-4 toEthernet fabric 220-3 is depicted by ODU path 1205. The reception ofEthernet packets 828 at egress PIU module 204-5 is depicted by workingODU path 1206-1. The reception of Ethernet packets 828 at egress PIUmodule 204-6 is depicted by protection ODU path 1207-1.

their arrival at egress PIU module 204-5, and ODU path 1207-1representing the arrival of the same Ethernet packets 828 at egress PIUmodule 204-6.

A multicast MAC address may be assigned and managed for each pair ofegress PIU modules 204, e.g. egress PIU modules 204-5 and 204-6, where afirst portion of the multicast MAC address may be assigned the virtualslot address associated with the corresponding Ethernet switch ports1116 of Ethernet switches 212 of Ethernet fabric 220-3 that areconnected to egress PIU module 204-5, previously described (see FIG.11). A second portion of the multicast MAC address may be assigned thevirtual slot address associated with the corresponding Ethernet switchports 1116 of Ethernet switches 212 of Ethernet fabric 220-3 that areconnected to egress PIU module 204-6, described above (see FIG. 11). Inan embodiment, the higher virtual slot number of egress PIU modules 204is assigned to the lower eleven bits of the multicast MAC address andthe lower virtual slot number is assigned to the higher eleven bits ofthe multicast MAC address. While the 22 bit space represent a fourmillion multicast MAC address space, the actual multicast MAC addressspace used may be smaller, because the actual number of egress PIUmodule pairs 204 that serve as egress PIU modules for 1+1 bridges may befairly small in practice and involves only a small set of coherentegress PIU module pairs. Because the actual multicast MAC address spacemay be small, the size of an Ethernet fabric MAC address table may besmall and may enable fast access to Ethernet fabric MAC address tableentries of the Ethernet fabric MAC address table. The management of anEthernet fabric MAC address table by an OTN switch network elementcontroller 214 (see FIG. 2) may also be more efficient by utilizingsimpler data structures, e.g. a linked list of Ethernet fabric MACaddress table entries. The smaller size of the Ethernet fabric MACaddress table and the utilization of the simpler data structures mayalso allow for the more efficient addition or removal of a Ethernetfabric MAC address table entries when a protection group is set up ortorn down. A multicast MAC address may be provisioned when a bridge, forexample, Ethernet fabric 220-3, indicates creation of the multicast MACaddress for transmissions to more than one egress PIU module 204. Oncethe multicast MAC address is created, all bridges that use the same pairof egress PIU modules 204 may share the same multicast MAC address.

In another embodiment, a multicast MAC address may be assigned andmanaged for each working ODU path 1206 and each protection path 1207that may utilize a bridge. An eight million multicast MAC address spaceis large enough to assign each working ODU path and protection path 1207a unique multicast MAC address in OTN switching system 1200 and toassign multicast MAC addresses that may help avoid any well-knownmulticast MAC addresses. In this embodiment, because the multicast MACaddress space is large, the size of the Ethernet fabric MAC addresstable may be large and OTN switch network element controller 214 mayutilize more complex data structures to enable more efficient access ofthe Ethernet fabric MAC address table.

Upon arrival of Ethernet packets 828, egress PIU module 204-5 convertsEthernet packets 828 to corresponding in sequence ODUs 834 and transmitsthem to ingress PIU module 204-7 via OTN network 1204-2, depicted byworking ODU path 1206-1. Upon arrival of the same Ethernet packets 828,egress PIU module 204-6 converts Ethernet packets 828 to the samecorresponding in sequence ODUs 834 and transmits them to ingress PIUmodule 204-8 via OTN network 1204-2, depicted by protection ODU path1207-1.

Tail end OTN switch 1202 may include an ingress processor 1210-1associated with ingress PIU module 204-7, an ingress processor 1210-2associated with ingress PIU module 204-8, and an egress processor 1210-3associated with egress PIU module 204-9. Ingress PIU module 204-7 mayinclude ingress processor 1210-1, ingress PIU module 204-8 may includeingress processor 1210-2, and egress PIU module 204-9 may include egressprocessor 1210-3. In one or more embodiments, ingress processor 1210-1may be included in a same PIU blade chassis 202 (see FIG. 2) as ingressPIU module 204-7, ingress processor 1210-2 may be included in a same PIUblade chassis 202 as ingress PIU module 204-8, and egress processor1210-3 may be included in a same PIU blade chassis 202 as egress PIUmodule 204-9.

During uni-directional SNCP switching operation of tail end OTN switch1202, ingress processor 1210-1 may monitor working ODU path 1206-1 todetect at least one of: a fault condition associated with working ODUpath 1206-1; an OTN command to perform a protection switch; a cessationof ODU transmission over working ODU path 1206-1; an impairment of ODUtransmission over working ODU path 1206-1, among other conditions.Ingress processor 1210-1 may analyze a detected condition to determinethe status of working ODU path 1206-1. Ingress processor 1210-2 maymonitor protection ODU path 1207-1 to detect at least one of: a faultcondition associated with protection ODU path 1207-1; an OTN command toperform the protection switch; an impairment of ODU transmission overprotection ODU path 1207-1; and an expiration of a keep alive delaytimer associated with protection ODU path 1207-1. Ingress processor1210-2 may analyze a detected condition to determine the status ofprotection ODU path 1207-1. Ingress processors 1210-1 and 1210-2 maycommunicate the respective status of working ODU path 1206-1 andprotection ODU path 1207-1 to each other and egress processor 1210-3.Ingress processors 1210-1 and 1210-2, and egress processor 1210-3 mayalso communicate various commands between them. In one or more otherembodiments, ingress PIU module 204-7 and ingress PIU module 204-8 maymonitor respective working ODU path 1206-1 and protection ODU path1207-1, and may communicate the respective status of working ODU path1206-1 and protection ODU path 1207-1 to each other and egress PIUmodule 204-9. Ingress PIU module 204-7, ingress PIU module 204-8, andegress PIU module 204-9 may also communicate various commands betweenthem.

Upon arrival of in sequence ODUs 834 at ingress PIU module 204-7,operating in the working state, ingress PIU module 204-7 converts ODUs834 to corresponding Ethernet packets 828 including the status ofworking ODU path 1206-1. Ingress PIU module 204-7 transmits Ethernetpackets 828 to egress PIU module 204-9 via Ethernet fabric 220-4,depicted by working ODU path 1206-1. Upon arrival of Ethernet packets828 at egress PIU module 204-9, Ethernet packets 828 may be stored atone of re-sequencing buffers 870 (see FIG. 10) of egress PIU module204-9 for further processing. Egress PIU module 204-9 may utilize thestatus in Ethernet packets 828 to monitor the state of working ODU path1206-1, described in further detail below.

A keep alive Ethernet packet having a keep alive sequence numberassociated with a protection ODU path 1207 may be transmittedperiodically, e.g. every N milliseconds, to report the state of theprotection ODU path 1207 and any commands and administrative actions onthe protection ODU path 1207. When the status of a protection ODU path1207 changes, a keep alive Ethernet packet may be transmittedimmediately. In other cases, several keep alive messages may be bundledinto a single keep alive Ethernet packet to reduce the number oftransmissions of keep alive Ethernet packets. An ingress PIU module 204may transmit keep alive Ethernet packet T consecutive times with thesame keep alive sequence number associated with the protection ODU path1207 which may prevent Ethernet packet loss, where T is greater than orequal to 1. The transmission of the keep alive Ethernet packets mayreduce Ethernet fabric 220-4 communication traffic. The keep alivesequence number associated with the protection ODU path 1207 may beincremented at the next time period the keep alive Ethernet packet istransmitted, where the starting value of the keep alive sequence numberassociated with the protection ODU path 1207 may be a random number. Theuse of keep alive sequence numbers may help minimize stall messages inthe OTN switching system 1200. Egress PIU module 204-9 may utilize thestatus of protection ODU path 1207-1 in the keep alive Ethernet packetsto monitor the state of protection ODU path 1207-1, described in furtherdetail below.

Ingress PIU modules 204-7 and 204-8 each have an associated hold offdelay associated with respective working ODU path 1206-1 and protectionODU path 1207-1 that may be managed by respective ingress PIU modules204-7 and 204-8. Each hold off delay may be utilized as part of thestate of respective working ODU path 1206-1 and protection ODU path1207-1. In one or more other embodiments, each hold off delay may bemanaged by respective working ingress processor 1210-1 and protectionprocessor 1210-2. In one or more other embodiments, a hold off delay maybe implemented using a timer, a delay device, or another mechanism.

Each protection ODU path 1207 may have an associated keep alive delaythat may be managed by an egress PIU module 204, where each keep alivedelay may be reset each time the egress PIU module 204 receives a keepalive Ethernet packet associated with a respective protection ODU path1207. For example, egress PIU module 204-9 may reset the keep alivedelay associated with protection ODU path 1207-1 upon receiving a keepalive Ethernet packet associated with protection ODU path 1207-1indicating that a condition has not been detected on protection ODU path1207-1. In one or more other embodiments, a keep alive delay may beimplemented using a timer, a delay device, or another mechanism.

Upon arrival of the same in sequence ODUs 834 at ingress PIU module204-8, operating in the protection state, ingress PIU module 204-8utilizes protection ODU path 1207-1 to transmit the status of protectionODU path 1207-1 in the keep alive Ethernet packets having a keep alivesequence number associated with protection ODU path 1207-1 to egress PIUmodule 204-9 via Ethernet fabric 220-4 using protection ODU path 1207-1.Ingress PIU module 204-8, operating in the protection state, maytransmit the keep alive Ethernet packets instead of transmittingEthernet packets 828 corresponding to the same in sequence ODUs 834.

In one or more other embodiments, ingress PIU module 204-8, operating inthe protection state, may convert the ODUs 834 to corresponding Ethernetpackets 828 including the status of protection ODU path 1207-1 and maytransmit Ethernet packets 828 to egress PIU module 204-9 via protectionODU path 1207-1. Transmitting the same Ethernet packets 828 to egressPIU module 204-9 may improve protection switching performance, thoughadditional Ethernet fabric hardware for transmission of both the workingODU path traffic and the protection ODU path traffic may be needed tomeet the Ethernet fabric bandwidth and frequency goals. In otherembodiments, egress PIU module 204-9 may receive the same Ethernetpackets 828 over protection ODU path 1207-1 that may further facilitatefaster protection switching performance although egress PIU module 204-9may need additional re-sequencing buffers 870 (see FIG. 10) to store thesame Ethernet packets 828 from ingress PIU module 204-7. In certainapplications, a trade-off between faster protection switchingperformance and additional Ethernet fabric hardware, e.g. additionalre-sequencing buffers 870, may be evaluated to determine a desiredoperation of OTN switch 1202.

During uni-directional SNCP switching operation of tail end OTN switch1202, an event or condition may be detected at one or more of ingressPIU module 204-7 and ingress PIU module 204-8. An event or a conditionmay be at least one of: a fault condition associated with a working ODUpath 1206, a fault condition associated with a protection ODU path 1207,an OTN command to perform a protection switch, a cessation of ODUtransmission over a working ODU path 1206, an impairment of ODUtransmission over a working ODU path 1206, an expiration of a hold offdelay associated with a respective working ODU path 1206 or protectionODU path 1207, among other events or conditions. Each event or conditionmay have an associated priority that indicates the priority of the eventor condition relative to the priorities of the other events orconditions. For example, a particular fault condition associated with aparticular working ODU path 1206 may have a priority that is higher thananother particular fault condition associated with another particularprotection ODU path 1207. For another example, a particular OTN commandto perform a protection switch may have a higher priority than aparticular fault condition associated with a particular protection ODUpath 1207. Ingress processor 1210-1 may determine the status of theworking ODU path 1206 based on the particular event or conditiondetected. When the event or condition is detected at ingress PIU module204-7, operating in the working state, ingress PIU module 204-7 sets thestatus of the working ODU path 1206 of Ethernet packets 828 to indicatethe event or condition prior to Ethernet packets 828 being transmittedto egress PIU module 204-9. Ingress processor 1210-2 may determine thestatus of the protection ODU path 1207 based on the event or conditiondetected. When the event or condition is detected at ingress PIU module204-8, operating in the protection state, ingress PIU module 204-8transmits a keep alive Ethernet packet having the status of theprotection ODU path 1207 set to indicate that the event or condition wasdetected at ingress PIU module 204-8.

Upon arrival of one or more of Ethernet packets 828 having the status ofworking ODU path 1206-1 and a keep alive Ethernet packet having thestatus of protection ODU path 1207-1, egress PIU module 204-9 maydetermine that a protection switch is to be performed on working ODUpath 1206-1 using protection ODU path 1207-1. The determination that theprotection switch may be performed may include egress PIU module 204-9determining that: an event or condition is detected at ingress PIUmodule 204-7, or an event or condition is detected at ingress PIU module204-8. The determination that the protection switch may be performed mayfurther include egress PIU module 204-9 determining that: a priority ofa particular working ODU path 1206-1 is higher than a priority of aparticular protection ODU path 1207-1.

In response to the determination, egress PIU module 204-9 performs theprotection switch to begin receiving Ethernet packets 828 from ingressPIU module 204-8 via protection ODU path 1207-1 and keep alive Ethernetpackets from ingress PIU module 204-7 via working ODU path 1206-1, asexplained in further detail below. In one or more other embodiments,egress processor 1210-3, performs the protection switch for egress PIUmodule 204-9 to begin receiving Ethernet packets 828 from ingress PIUmodule 204-8 via protection ODU path 1207-1 and the correspondingEthernet packets 828 from ingress PIU module 204-7 via working ODU path1206-1.

Egress PIU module 204-9, as part of the protection switch, may transmitone or more stop ODU transmission messages having an assigned stop ODUtransmission sequence number associated with working ODU path 1206-1 toingress PIU module 204-7. Egress PIU module 204-9 may also transmit oneor more start ODU transmission messages having an assigned start ODUtransmission sequence number associated with protection ODU path 1207-1to ingress PIU module 204-8. Egress PIU module 204-9 may further drainEthernet packets 828 received over working ODU path 1206-1 from therespective re-sequencing buffer 870 associated with egress PIU module204-9. Egress PIU module 204-9 may also fill the respectivere-sequencing buffer 870 associated with egress PIU module 204-9 withEthernet packets 828 received over protection ODU path 1207-1. EgressPIU module 204-9 may also re-establish clock synchronization withprotection ODU path 1207-1. Egress PIU module 204-9 may drain therespective re-sequencing buffer 870 by flushing the re-sequencing buffer870 and discarding any Ethernet packets 828 received from working ODUpath 1206-1.

Each of the stop and start ODU transmission messages may be sent anumber T consecutive times to minimize Ethernet packet loss, where thevalue of T is greater than or equal to 1. In an embodiment, T is set to3. In one or more other embodiments, the value of T may be set during aconfiguration of tail end OTN switch 1202, a system boot up, or thelike. Ingress PIU module 204-7, in response to receiving the one or morestop ODU transmission messages, starts operating in the protection stateand transmits keep alive Ethernet packets over working ODU path 1206-1.Ingress PIU module 204-8, in response to receiving the one or more startODU transmission messages, starts operating in the working state andtransmits Ethernet packets 828 over protection ODU path 1206-3.

Operating OTN switching system 1200 in this manner, provides sub-networkconnection protection (SNCP) of working ODU paths 1206. Tail end OTNswitch 1202 may protect a working ODU path 1206 by establishing aredundant protection ODU path 1207 to protect the working ODU path 1206.Tail end OTN switch 1202 may detect an event or condition associatedwith the working ODU path 1206 and may determine that a protectionswitch may be performed on the working ODU path 1206 based on thedetection of the event or condition. In response to the determination,tail end OTN switch 1202 performs the protection switch using the usingthe protection ODU path 1207, which provides 1+1 OTN protection.

Referring now to FIG. 13, a block diagram of an example of concatenationof ODU path protection in an embodiment of an OTN switching system 1300is illustrated. In FIG. 13, OTN switching system 1300 is shown in aschematic representation and is not drawn to scale or perspective. It isnoted that, in different embodiments, OTN switching system 1300 may beoperated with additional or fewer elements.

In OTN switching system 1300, head end OTN switch 1201 and tail end OTNswitch 1202 operate as previously described with reference to FIG. 12.In OTN switching system 1300 shown in FIG. 13, a concatenation point OTNswitch 1301 has been added in between head end OTN switch 1201 and tailend OTN switch 1202. Concatenation point OTN switch 1301 operatessimilar to the operation of tail end OTN switch 1202 but may alsoprevent a failure in the OTN network 1204-4 SNCP domain from causingsympathetic protection switching in the OTN network 1204-5 SNCP domainwhile in subnetwork connection with non-intrusive end-to-end monitoring(SNC/Ne). A sympathetic protection switch is a protection switch causedby failure propagation into the intended protection domain, e.g. OTNnetwork 1204-5 SNCP domain, from outside the intended protection domain,e.g. the OTN network 1204-4 domain. The hold off delays in the OTNnetwork 1204-5 SNCP domain may be configured to have longer hold offdelay expiration values than the hold off delay expiration valuesdescribed above with reference to FIG. 12. As shown, 1+1 SNCP protectionfor concatenation point OTN switch 1301 is similar to the 1+1 protectionfor tail end OTN switch 1202.

In concatenation point OTN switch 1301, a single multicast MAC addressmay be utilized as in the 1+1 protection for OTN switching system 1200of FIG. 12. Working ODU path 1206-2 and protection ODU path 1207-2 mayutilize the single multicast MAC address, where ingress PIU module204-10, operating in the working state, uses the single multicast MACaddress to transmit Ethernet packets 828 to both egress PIU modules204-12 and 204-13. Ingress PIU module 204-11, operating in theprotection state, may refrain from transmitting the correspondingEthernet packets 828 to Ethernet fabric 220-5, for example to reducetraffic and increase performance of Ethernet fabric 220-5.

In FIG. 13, the protection switch may be initiated, in one embodiment,by egress PIU module 204-12 on working ODU path 1206-2. Concatenationpoint OTN switch 1301 may perform the protection switch for: egress PIUmodule 204-12 to receive Ethernet packets 828 from ingress PIU module204-11 via protection ODU path 1207-2 and egress PIU module 204-13 toreceive the corresponding Ethernet packets 828 from ingress PIU module204-11 via protection ODU path 1207-2.

In FIG. 13, the protection switch may be initiated, in some embodiments,by egress PIU module 204-12 on working ODU path 1206-2. Thecommunication between egress PIU module 204-12 and ingress PIU modules204-10 and 204-11 is the same as in 1+1 SNCP protection switchingpreviously described with reference to FIG. 12, with an additionalcommunication between egress PIU module 204-12 and egress PIU module204-13. In one or more other embodiments, the protection switch may beinitiated by egress processor 1210-6, and egress processor 1210-6,ingress processor 1210-4, ingress processor 1210-5, and egress processor1210-7 may communicate status, various OTN commands, and otherinformation and data between each other. Egress processor 1210-7 mayperiodically monitor egress processor 1210-6 for an indication thategress processor 1210-6 has a fault condition. Egress processor 1210-7,in response to the indication that egress processor 1210-6 has the faultcondition, will take over the monitoring and protection switchinitiation function from egress processor 1210-6. In one or more otherembodiments, egress PIU module 204-13 may similarly periodically monitoregress PIU module 204-12 for an indication that egress PIU module 204-12has a fault condition. Egress PIU module 204-13, in response to theindication that egress PIU module 204-12 has the fault condition, willtake over the monitoring and protection switch initiation function fromPIU module 204-12.

Referring now to FIG. 14, a block diagram of an example embodiment of anOTN switching system 1400 is illustrated. In FIG. 14, OTN switchingsystem 1400 is shown in a schematic representation and is not drawn toscale or perspective. It is noted that, in different embodiments, OTNswitching system 1400 may be operated with additional or fewer elements.In OTN switching system 1400, subnetwork connection protection withinherent monitoring (SNC/I) on high order ODU may be used to supportOTU-link layer protection, which may be utilized for inter-domainprotected hand-offs.

During operation of concatenation point OTN switch 1401, ingress PIUmodule 204-14, operating in the working state, may transmit Ethernetpackets 828 associated with working ODU path 1206-3 to egress PIU module204-16. Ingress PIU module 204-14 may also transmit Ethernet packets 828associated with working ODU path 1206-4 to egress PIU module 204-17.Ingress PIU module 204-14 may further transmit Ethernet packets 828associated with working ODU path 1206-5 to egress PIU module 204-18, viaEthernet fabric 220-6. Ingress PIU module 204-15, operating in theprotection state, may refrain from transmitting the correspondingEthernet packets 828 associated with ODU path 1207-3, ODU path 1207-4,and ODU path 1206-5.

During operation of concatenation point OTN switch 1401, at least one ofingress PIU module 204-14, operating in the working state, and ingressPIU module 204-15, operating in the protection state, may determine thata protection switch may be performed. In response to the determination,egress PIU modules 204-16, 204-17, and 204-18 may be informed. Thecommunication between ingress PIU modules 204-14 and 204-15, and egressPIU modules 204-16, 204-17, and 204-18 is similar to the communicationdescribed above with reference to FIG. 12 and FIG. 13. In one or moreother embodiments, two or more ingress PIU modules 204 may determinethat a protection switch may be performed. In response to thedetermination, two or more egress PIU modules 204 may be informed. Inone or more other embodiments, ingress processors 1210-8 and 1210-9 maydetermine that a protection switch may be performed, and ingressprocessors 1210-8 and 1210-9, and egress processors 1210-10, 1210-11,and 1210-12 may communicate status, various OTN commands, and otherinformation and data between each other. Concatenation point OTN switch1401 may perform the protection switch for egress PIU module 204-16 toreceive Ethernet packets 828 associated with protection ODU path 1207-3from ingress PIU module 204-15, via Ethernet fabric 220-6. In addition,egress PIU module 204-17 may receive Ethernet packets 828 associatedwith protection ODU path 1207-4 from ingress PIU module 204-15, viaEthernet fabric 220-6. Further, egress PIU module 204-18 may receiveEthernet packets 828 associated with protection ODU path 1207-5 fromingress PIU module 204-15, via Ethernet fabric 220-6.

Referring now to FIG. 15, a block diagram of an example of Ethernetfabric protection in an embodiment of an OTN switching system 1500 isillustrated. In FIG. 15, OTN switching system 1500 is shown in aschematic representation and is not drawn to scale or perspective. It isnoted that, in different embodiments, OTN switching system 1500 may beoperated with additional or fewer elements.

In FIG. 15, a PIU module 204 of a plurality of PIU modules 204 maydetect a fault condition 1522 on an Ethernet fabric plane 1520 of anEthernet fabric 220-7. In response to the detection, OTN switchingsystem 1500 may transmit the fault condition 1522 to other PIU modules204 of the plurality of PIU modules 204 to redirect ODU traffic awayfrom the fault on the Ethernet fabric plane 1520. By redirecting the ODUtraffic away from the fault, OTN switching system 1500 protects theEthernet fabric 220-7, which allows for continuous transmission of theODU traffic.

In OTN switching system 1500, a particular PIU module 204 may detect atleast one of: a fault condition 1522 associated with a failure of one ofthe Ethernet fabric planes 1520, a fault condition 1522 associated withan administrative action to shut down one of the Ethernet fabric planes1520, a fault condition 1522 associated with a failure of one of PIUports 208 of the particular PIU module 204, a fault condition 1522associated with a failure of a connection between one of PIU ports 208of the particular PIU module 204 and a corresponding Ethernet switch212, among other fault conditions 1522. OTN switching system 1500 mayhave the capability to provide sufficient bandwidth to allow forcontinuous transmission of the ODU traffic in the case of: one or moreEthernet fabric planes 1520 fails or is shutdown; one or more PIU ports208 of corresponding PIU modules 204 fails, one or more connections tocorresponding one or more PIU ports 208 of corresponding PIU modules 204fail, among other failures. For example, OTN switching system 1500provides 1:3 Ethernet fabric protection when one of the Ethernet fabricplanes 1520 of Ethernet fabric 220-7 fails.

During operation of OTN switching system 1500, a failure of an Ethernetfabric plane 1520-1 of an Ethernet switch 212-9 may occur. That failuremay or may not be immediately detected by the Ethernet switch 212-9itself. However, it may be detected by its communication counterparts,which are PIUs 204. A PIU module 204-19 may detect a fault condition1522-1 associated with the failure of Ethernet fabric plane 1520-1 byseeing its failure condition 1522-2. At this moment, PIU 204-19 and itsassociated blade controller 1514-1 may not always know if the failure isa localized link failure or a wider-ranging failure on Ethernet switch212-9. Before 1514-1 detects or receives any indication that there maybe wider failure than a local link failure, it treats the failure as alocal link failure by stopping ODU traffic forwarding towards the failedlink but not the whole Ethernet forwarding plane. PIU module 204-19 mayalso stop transmission of ODU traffic via Ethernet fabric plane 1520-1,while continuing to transmit ODU traffics on the rest of the Ethernetfabric planes. PIU module 204-19 may further communicate fault condition1522-1 to an associated PIU blade controller 1514-1 of PIU blade chassis220-3. PIU blade controller 1514-1 may communicate fault condition1522-1 to other associated PIU modules 204 so that the other associatedPIU modules 204, e.g. PIU module 204-20, may stop transmission of ODUtraffic to the PIU module 204 detecting the failure (in this case PIUmodule 204-19) via Ethernet fabric plane 1520-1, while continuing totransmit ODU traffics to PIU module 204-19 on the rest of the Ethernetfabric planes. PIU module 204-20 (before detecting its own link failure1522-3 in this specific example of FIG. 15) may continue to transportother ODU traffic over all Ethernet fabric planes, including 1520-1. Afailure message having a sequence number associated with fault condition1522-1 may be transmitted immediately to report fault condition 1522-1to other PIU blade controllers 1514 associated with other PIU modules204 using any one of the other Ethernet fabric planes 1520-2 through1520-4. The failure message may be transmitted as a MAC broadcastreal-time control message. PIU blade controller 1514-1 may transmit thefailure message T consecutive times with the same sequence numberassociated with fault condition 1522-1, where T is greater than or equalto 1. PIU blade controller 1514-1 may transmit the failure message usinga cyclical walk sequence 944 (see FIG. 9), previously described. PIUblade controller 1514-1 transmitting the failure message to the otherPIU blade controllers 1514 may reduce the number of MAC broadcastreal-time control messages. In one or more other embodiments, PIU module204-19 may transmit the failure message associated with fault condition1522-1 to report fault condition 1522-1 to the other PIU modules 204.

Upon receiving the failure message associated with fault condition1522-1, the other PIU blade controllers 1514 may communicate faultcondition 1522-1 to the other associated PIU modules 204 so that theother associated PIU modules 204 may stop transmission of ODU traffic tothe PIU module that detected the fault (PIU module 204-19) via Ethernetfabric plane 1520-1, while continuing to transmit ODU traffics on therest of the Ethernet fabric planes. As illustrated in FIG. 15, uponreceiving the failure message associated with fault condition 1522-1,PIU blade controller 1514-2 may communicate fault condition 1522-1 toPIU module 204-21 and PIU module 204-22 so that PIU module 204-21 andPIU module 204-22 may stop their transmission of ODU traffic to PIUmodule 204-19 via Ethernet fabric plane 1520-1, while continuing totransmit ODU traffics on the rest of the Ethernet fabric planes.

In one or more other embodiments, during operation of OTN switchingsystem 1500, fault condition 1522-2 associated with the failure of theith PIU port 208-13 of corresponding PIU module 204-19 on the ithEthernet fabric plane 1520-1 and a second fault condition 1522-3associated with a failure of an ith PIU port 208-17 of corresponding PIUmodule 204-20 on the ith Ethernet fabric plane 1520-1 may occur. Thefault condition 1522-2 and the second fault condition 1522-3 occur atsubstantially the same time. PIU module 204-19 may detect the faultcondition 1522-2 and may stop transmission of its respective ODU trafficvia its ith PIU port 208-13. PIU module 204-19 may also communicate thefault condition 1522-2 to PIU blade controller 1514-1. PIU module 204-20may detect the second fault condition 1522-3 and may stop transmissionof its respective ODU traffic via its ith PIU port 208-17. PIU module204-20 may also communicate the second fault condition 1522-3 to PIUblade controller 1514-1. At this moment, PIU blade controller 1514-1understands the failure is beyond a local link failure 1522-2 originallydetected PIU 204-19. Now PIU blade controller 1514-1 may instruct itsPIU modules 204 not to use the Ethernet fabric plane 1520-1 for any ODUtransmission. PIU blade controller 1514-1 may communicate faultcondition 1522-2 to PIU module 204-20 so that PIU module 204-20 may stoptransmission of its respective ODU traffic to the ith PIU port 208-13 ofcorresponding PIU module 204-19. PIU blade controller 1514-1 may alsocommunicate fault condition 1522-3 to PIU module 204-19 so that PIUmodule 204-19 may stop transmission of its respective ODU traffic to theith PIU port 208-17 of corresponding PIU module 204-20. As previouslydescribed, a MAC broadcast failure message associated with the faultcondition 1522-2 and the second fault condition 1522-3 may be similarlytransmitted immediately to report the fault condition 1522-2 and thesecond fault condition 1522-3 to PIU blade controller 1514-2 associatedwith PIU modules 204-21 and 204-22 using any one of Ethernet fabricplanes 1520-2 through 1520-4.

Upon receiving the failure message associated with the fault condition1522-2 and the second fault condition 1522-3, PIU blade controller1514-2 also understands it is more than a local link failure andtherefore may stop transmitting ODU traffic over Ethernet fabric plane1520-1. PIU blade controller 1514-2 may communicate the fault condition1522-2 and the second fault condition 1522-3 to PIU modules 204-21 and204-22 so that PIU modules 204-21 and 204-22 may stop transmission oftheir respective ODU traffic to the ith PIU port 208-13 of correspondingPIU module 204-19 and the ith PIU port 208-17 of corresponding PIUmodule 204-20 on the ith Ethernet fabric plane 1520-1. Handling theoccurrence of the fault condition 1522-2 and the second fault condition1522-3 in this manner may allow for a simpler mechanism to provideEthernet fabric protection.

In one or more other embodiments, during operation of OTN switchingsystem 1500, the fault condition 1522-2 associated with the failure ofthe ith PIU port 208-13 of corresponding PIU module 204-19 on the ithEthernet fabric plane 1520-1 and the second fault condition 1522-3associated with the failure of the ith PIU port 208-17 of correspondingPIU module 204-20 on the ith Ethernet fabric plane 1520-1 may occur. Thefault condition 1522-2 and the second fault condition 1522-3 occur atsubstantially the same time. PIU module 204-19 may detect the faultcondition 1522-2 and may communicate the fault condition 1522-2 to PIUblade controller 1514-1. PIU module 204-20 may detect the second faultcondition 1522-3 and may communicate the second fault condition 1522-3to PIU blade controller 1514-1. PIU blade controller 1514-1 maycommunicate fault condition 1522-2 to PIU module 204-20 so that PIUmodule 204-20 may stop transmission of its respective ODU traffic viaall its PIU ports 208-17 through 208-20. PIU blade controller 1514-1 mayalso communicate fault condition 1522-3 to PIU module 204-19 so that PIUmodule 204-19 may stop transmission of its respective ODU traffic viaall its PIU ports 208-13 through 208-16. PIU blade controller 1514-1 mayfurther create a MAC broadcast failure message associated with the faultcondition 1522-2 and the second fault condition 1522-3.

PIU blade controller 1514-1 may start a delay associated with PIU module204-19 and PIU module 204-20. In an embodiment, the delay may be set toa random number between 0 and N. For example, N may be set to 100 microseconds. In the case that the delay has expired without PIU bladecontroller 1514-1 receiving any failure messages from any other PIUblade controller 1514, e.g. PIU blade controller 1514-2, PIU bladecontroller 1514-1 may transmit a first MAC broadcast failure messageassociated with the fault condition 1522-2 and the second faultcondition 1522-3 immediately to report the fault condition 1522-2 andthe second fault condition 1522-3 to PIU blade controller 1514-2associated with PIU modules 204-21 and 204-22 PIU blade controller1514-2. After transmitting the first MAC broadcast message, PIU bladecontroller 1514-1 may start a second delay associated with PIU module204-19 and PIU module 204-20. In an embodiment, the second delay may beset to a number equal to (2*N+M). For example, N may be set as above andM may be set to a value less than 1000 micro seconds.

After expiration of the second delay without PIU blade controller 1514-1receiving any failure messages from PIU blade controller 1514-2, PIUblade controller 1514-1 is the only one detecting multiple ith PIU port208 failures on the ith Ethernet fabric plane 1520-1. In this case, PIUblade controller 1514-1 may transmit a second MAC broadcast failuremessage associated with the fault condition 1522-2 and the second faultcondition 1522-3 immediately to report the fault condition 1522-2 andthe second fault condition 1522-3 to PIU blade controller 1514-2associated with PIU modules 204-21 and 204-22. PIU blade controller1514-1 may communicate to PIU module 204-19 and PIU module 204-20 sothat PIU module 204-19 and PIU module 204-20 may start transmission ofODU traffic via all its non-failing PIU ports 208-14 through 208-16 andnon-failing PIU ports 208-18 through 208-20 respectively.

A fault condition 1522-4 associated with the failure of the jth PIU port208-22 of corresponding PIU module 204-21 on the jth Ethernet fabricplane 1520-2 has been detected by PIU module 204-21. In the case thatPIU blade controller 1514-1 has received a third MAC broadcast failuremessage associated with the fault condition 1522-4 from PIU bladecontroller 1514-2 prior to the expiration of the second delay, PIU bladecontroller 1514-1 may create a PIU port status map that indicates whichPIU ports 208 on the ith Ethernet fabric plane 1520-1 and on the jthEthernet fabric plane 1520-2 have failed. After expiration of the seconddelay, PIU blade controller 1514-1 may start transmission of therespective ODU traffic from the M PIU ports 208-14 through 208-16 of PIUmodule 204-19, other than the ith PIU port 208-13 of PIU module 204-19,to the PIU modules 204-20, 204-21 and 204-22, other than the ith PIUport 208-17 of PIU module 204-20, and the jth PIU port 208-22 of PIUmodule 204-21. PIU blade controller 1514-1 may also start transmissionof the respective ODU traffic from the M PIU ports 208-18 through 208-20of PIU module 204-20, other than the ith PIU port 208-17 of PIU module204-20, to the PIU modules 204-19, 204-21 and 204-22, other than the ithPIU port 208-13 of PIU module 204-19, and the jth PIU port 208-22 of PIUmodule 204-21. PIU blade controller 1514-1 may use the PIU port statusmap to determine which associated non-failing PIU ports 208-13 through208-16 of PIU module 204-19 and non-failing PIU ports 208-17 through208-20 of PIU module 204-20 may be used for transmission of theirrespective ODU traffic and which failing PIU ports 208-21 through 208-24of PIU module 204-21 and failing PIU ports 208-25 through 208-28 of PIUmodule 204-22 should not be transmitted the respective ODU traffic. PIUblade controller 1514-1 may also transmit a fourth MAC broadcast failuremessage associated with the fault condition 1522-2, the second faultcondition 1522-3, and the third fault condition 1522-4 immediately toreport the fault condition 1522-2, the second fault condition 1522-3,and the third fault condition 1522-4 to PIU blade controller 1514-2associated with PIU modules 204-21 and 204-22. PIU blade controller1514-1 may further transmit its PIU port status map associated with thefault condition 1522-2, the second fault condition 1522-3, and the thirdfault condition 1522-4 to OTN switch controller 214.

In the case that PIU blade controller 1514-1 has received failuremessages from other PIU blade controllers 1514, e.g. PIU bladecontroller 1514-2, prior to the expiration of the delay, it may indicatethat Ethernet fabric plane 1520-1 has failed. After expiration of thedelay, PIU blade controller 1514-1 may start the second delay and athird delay associated with PIU module 204-19 and PIU module 204-20. Inan embodiment, the third delay may be set to a random between N/2 and M.The second delay, the number N, and the number M are as previouslydescribed.

After expiration of the second delay, PIU blade controller 1514-1 maytransmit a fifth MAC broadcast failure message associated with the PIUport status map that indicates which PIU ports 208 on the M Ethernetfabric planes 1520 have failed. The current PIU port status mapindicates that the ith PIU port 208-13 of corresponding PIU module204-19 on the ith Ethernet fabric plane 1520-1 and the ith PIU port208-17 of corresponding PIU module 204-20 on the ith Ethernet fabricplane 1520-1 have failed. PIU blade controller 1514-1 may continue toupdate the PIU port status map from any failure messages received fromPIU blade controller 1514-2. After expiration of the third delay, PIUblade controller 1514-1 may start transmission of the respective ODUtraffic from the M PIU ports 208-14 through 208-16 of PIU module 204-19,other than the ith PIU port 208-13 of PIU module 204-19, to the PIUmodules 204-20, 204-21 and 204-22, other than the ith PIU port 208-17 ofPIU module 204-20. PIU blade controller 1514-1 may also starttransmission of the respective ODU traffic from the M PIU ports 208-18through 208-20 of PIU module 204-20, other than the ith PIU port 208-17of PIU module 204-20, to the PIU modules 204-19, 204-21 and 204-22,other than the ith PIU port 208-13 of PIU module 204-19. PIU bladecontroller 1514-1 may further transmit its PIU port status map to OTNswitch controller 214.

In one or more other embodiments, during operation of OTN switchingsystem 1500, PIU modules 204-21 and 204-22 associated with PIU bladecontroller 1514-2 may not have detected any fault conditions 1522associated with their own PIU ports 208-21 through 208-28 correspondingto respective PIU modules 204-21 and 204-22. Upon PIU blade controller1514-2 receiving a sixth MAC broadcast failure message associated withfault condition 1522-2 associated with PIU port 208-13 of correspondingPIU module 204-19 on the ith Ethernet fabric plane 1520-1, PIU bladecontroller 1514-2 may stop transmission of its respective ODU trafficvia all its PIU ports 208-21 of PIU module 204-21 and 208-25 of PIUmodule 204-22 on the ith Ethernet fabric plane 1520-1. PIU bladecontroller 1514-2 may also start an associated fourth delay. In anembodiment, the fourth delay may be set to (3×N+M), where N and M are aspreviously described. The delay, the second delay, the third delay, andthe fourth delay may each be implemented using a timer, a delay device,or another mechanism. The expiration of the third delay is after theexpiration of the second delay, and the expiration of the second delayis after the expiration of the delay. PIU blade controller 1514-2 mayalso create a second PIU port status map that indicates PIU port 208-13of corresponding PIU module 204-19 on the ith Ethernet fabric plane1520-1 have failed. PIU blade controller 1514-2 may further update thesecond PIU port status map based on any further MAC broadcast failuremessages it receives or fault conditions detected by PIU modules 204-21and 204-22.

After expiration of the fourth delay, PIU blade controller 1514-2 maystart transmission of the respective ODU traffic from the M PIU ports208-21 through 208-24 of PIU module 204-21 to the PIU modules 204-19,204-20 and 204-22, other than the ith PIU port 208-13 of PIU module204-19. PIU blade controller 1514-2 may also start transmission of therespective ODU traffic from the M PIU ports 208-25 through 208-28 of PIUmodule 204-22 to the PIU modules 204-19, 204-20 and 204-21, other thanthe ith PIU port 208-13 of PIU module 204-19. PIU blade controller1514-2 may further transmit a seventh MAC broadcast failure messageassociated with the second PIU port status map that indicates which PIUports 208 on the M Ethernet fabric planes 1520 have failed to the PIUports 208 of PIU modules 208 on the corresponding Ethernet fabric planes1520 that have not failed. The current second PIU port status mapindicates that the ith PIU port 208-13 of corresponding PIU module204-19 on the ith Ethernet fabric plane 1520-1 has failed. PIU bladecontroller 1514-2 may further transmit its second PIU port status map toOTN switch controller 214.

In one or more other embodiments, during operation of OTN switchingsystem 1500, PIU modules 204-21 associated with PIU blade controller1514-2 may have detected the fault condition 1522-4 associated with itsown PIU port 208-21. In this case, PIU blade controller 1514-2 performsthe same Ethernet fabric protection as the case in which PIU modules204-21 and 204-22 associated with PIU blade controller 1514-2 may nothave detected any fault conditions 1522 associated with their own PIUports 208-21 through 208-28 corresponding to respective PIU modules204-21 and 204-22.

OTN switch controller 214 may synthesize all the failure conditions fromall the PIU port status maps and failure messages it has received andanalyze all of the synthesized failures to determine if anydiscrepancies exist. When the determination indicates that a discrepancyexists, OTN switch controller 214 may raise an alarm and push a worstcase PIU port status map to all the PIU blade controllers 1514 in OTNswitching system 1500. When the determination indicates that adiscrepancy does not exist, OTN switch controller 214 may record all ofthe synthesized failure conditions and perform management tasks relatedto all of the synthesized failures for OTN switching system 1500.

By having the PIU blade controllers 1514 of OTN switching system 1500perform Ethernet fabric protection in this distributed manner, theefficiency of determining how many PIU ports 208 of corresponding PIUmodules 204 have failed and how many have not failed may be improved.Performing Ethernet fabric protection in this distributed manner, mayalso allow the PIU blade controllers on the same Ethernet fabric plane1520 to be made aware of the PIU port failures in real-time. PerformingEthernet fabric protection in this distributed manner, may also allowthe non-failing PIU ports 208 on the same Ethernet fabric plane 1520 tobe used so that all of the Ethernet fabric planes 1520 may continue tobenefit from load sharing while corrective action to repair or replacethe failing Ethernet fabric planes or PIU port failures.

Referring now to FIG. 16, a flowchart of selected elements of anembodiment of a method 1600 for Ethernet fabric protection in an OTNswitching system, as described herein, is depicted. In variousembodiments, method 1600 may be performed using OTN switching systems200, 1200, 1300, 1400, and 1500. It is noted that certain operationsdescribed in method 1600 may be optional or may be rearranged indifferent embodiments.

The OTN switching system of method 1600 may include an OTN switch. TheOTN switch may include an Ethernet fabric having a number M of Ethernetfabric planes, each of the M Ethernet fabric planes may include acorresponding Ethernet switch of M Ethernet switches. The OTN switch mayalso include a plurality of PIU modules each having M PIU portsincluding a first PIU module, where an ith PIU port of each of theplurality of PIU modules may be connected to the ith Ethernet switch ofthe ith Ethernet fabric plane of the Ethernet fabric. Method 1600 maybegin at step 1602, assigning a variable i having a value ranging from 1to M to denote the ith Ethernet fabric plane of the M Ethernet fabricplanes, the ith Ethernet switch of the M Ethernet switches, and the ithPIU port of the M PIU ports, where M is greater than one. At step 1604,detecting, by the first PIU module, a fault condition associated withthe ith PIU port of the first PIU module on the ith Ethernet fabricplane. At step 1606, transmitting the fault condition to stoptransmission of ODU traffic from the plurality of PIU modules to the ithPIU port of the first PIU module.

As disclosed herein, methods and systems for Ethernet fabric protectionin a disaggregated OTN switching system that include PIU modules eachhaving multiple ports for OTN to Ethernet transceiving and an Ethernetfabric as a switching core are disclosed. An OTN over Ethernet module ineach of the PIU modules may enable various OTN functionality to berealized using the Ethernet fabric which may include multiple Ethernetswitches. A first PIU module may detect a fault condition on an Ethernetfabric plane of the Ethernet fabric. In response to the detection, theOTN switching system may transmit the fault condition to other PIUmodules to redirect ODU traffic away from the fault on the Ethernetfabric plane.

The above disclosed subject matter is to be considered illustrative, andnot restrictive, and the appended claims are intended to cover all suchmodifications, enhancements, and other embodiments which fall within thetrue spirit and scope of the present disclosure. Thus, to the maximumextent allowed by law, the scope of the present disclosure is to bedetermined by the broadest permissible interpretation of the followingclaims and their equivalents, and shall not be restricted or limited bythe foregoing detailed description.

What is claimed is:
 1. A method for Ethernet fabric protection, in anoptical transport network (OTN) switch comprising: an Ethernet fabrichaving a number M of Ethernet fabric planes, each of the M Ethernetfabric planes includes a corresponding Ethernet switch of M Ethernetswitches; and a plurality of plug-in universal (PIU) modules each havingM PIU ports including a first PIU module, wherein an ith PIU port ofeach of the plurality of PIU modules is connected to the ith Ethernetswitch of the ith Ethernet fabric plane of the Ethernet fabric, themethod comprising: assigning a variable i having a value ranging from 1to M to denote the ith Ethernet fabric plane of the M Ethernet fabricplanes, the ith Ethernet switch of the M Ethernet switches, and the ithPIU port of the M PIU ports, wherein M is greater than one; detecting,by the first PIU module, a fault condition associated with the ith PIUport of the first PIU module on the ith Ethernet fabric plane; andtransmitting the fault condition to stop transmission of optical dataunit (ODU) traffic from the plurality of PIU modules to the ith PIU portof the first PIU module.
 2. The method of claim 1, further comprising:after transmitting the fault condition to stop transmission of the ODUtraffic from the plurality of PIU modules to the ith PIU port of thefirst PIU module, stopping the transmission of the ODU traffic from theplurality of PIU modules to the ith PIU port of the first PIU module. 3.The method of claim 1, further comprising: detecting, by a second PIUmodule of the plurality of PIU modules, a second fault conditionassociated with the ith PIU port of the second PIU module on the ithEthernet fabric plane; and transmitting the second fault condition tostop the transmission of the ODU traffic from the plurality of PIUmodules to the ith PIU port of the second PIU module.
 4. The method ofclaim 3, wherein transmitting the fault condition and transmitting thesecond fault condition are transmitted in the same transmission.
 5. Themethod of claim 1, further comprising: after receiving the faultcondition and a second fault condition associated with the ith PIU portof a second PIU module of the plurality of PIU modules on the ithEthernet fabric plane to stop the transmission of the ODU traffic fromthe plurality of PIU modules to the ith PIU port of the second PIUmodule, stopping the transmission of the ODU traffic from the pluralityof PIU modules to the ith PIU port of the first PIU module and the ithPIU port of the second PIU module.
 6. The method of claim 1, furthercomprising: prior to transmitting the fault condition to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the first PIU module and transmitting a second faultcondition associated with the ith PIU port of a second PIU module of theplurality of PIU modules on the ith Ethernet fabric plane to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the second PIU module, stopping the transmission of theODU traffic from the M PIU ports of the first PIU module and the M PIUports of the second PIU module; and after expiration of a delayassociated with the first PIU module and the second PIU module,transmitting the second fault condition to stop the transmission of theODU traffic from the plurality of PIU modules to the ith PIU port of thesecond PIU module.
 7. The method of claim 6, further comprising: afterexpiration of a second delay associated with the first PIU module andthe second PIU module, transmitting the ODU traffic from the M PIU portsof the first PIU module other than the ith PIU port of the first PIUmodule and transmitting the ODU traffic from the M PIU ports of thesecond PIU module other than the ith PIU port of the second PIU module,wherein the expiration of the second delay is after the expiration ofthe delay.
 8. The method of claim 6, further comprising: receiving athird fault condition associated with the jth PIU port of a third PIUmodule on the ith Ethernet fabric plane to stop the transmission of theODU traffic from the plurality of PIU modules to the jth PIU port of thethird PIU module; and after expiration of a second delay associated withthe first PIU module and the second PIU module, transmitting the ODUtraffic from the M PIU ports of the first PIU module, other than the ithPIU port of the first PIU module, and the M PIU ports of the second PIUmodule, other than the ith PIU port of the second PIU module, to theplurality of PIU modules other than the ith PIU port of the first PIUmodule, the ith PIU port of the second PIU module, and the jth PIU portof the third PIU module, wherein the expiration of the second delay isafter the expiration of the delay.
 9. The method of claim 1, furthercomprising: prior to transmitting the fault condition to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the first PIU module and transmitting a second faultcondition associated with the ith PIU port of a second PIU module of theplurality of PIU modules on the ith Ethernet fabric plane to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the second PIU module, stopping the transmission of theODU traffic from the M PIU ports of the first PIU module and the M PIUports of the second PIU module; receiving a third fault conditionassociated with the jth PIU port of a third PIU module on the ithEthernet fabric plane to stop the transmission of the ODU traffic fromthe plurality of PIU modules to the jth PIU port of the third PIUmodule; after expiration of a second delay associated with the first PIUmodule and the second PIU module, transmitting the second faultcondition to stop the transmission of the ODU traffic from the pluralityof PIU modules to the ith PIU port of the second PIU module; andtransmitting the third fault condition to stop the transmission of theODU traffic from the plurality of PIU modules to the jth PIU port of thethird PIU module.
 10. The method of claim 9, further comprising: afterexpiration of a third delay associated with the first PIU module and thesecond PIU module, transmitting the ODU traffic from the M PIU ports ofthe first PIU module, other than the ith PIU port of the first PIUmodule, and the M PIU ports of the second PIU module, other than the ithPIU port of the second PIU module, to the plurality of PIU modules otherthan the ith PIU port of the first PIU module, the ith PIU port of thesecond PIU module, and the jth PIU port of the third PIU module, whereinthe expiration of the third delay is after the expiration of the seconddelay.
 11. The method of claim 1, further comprising: after receivingthe fault condition and a second fault condition associated with the ithPIU port of a second PIU module of the plurality of PIU modules on theith Ethernet fabric plane to stop the transmission of the ODU trafficfrom the plurality of PIU modules to the ith PIU port of the second PIUmodule, stopping the transmission of the ODU traffic from the M PIUports of a third PIU module of the plurality of PIU modules; and afterexpiration of a delay associated with the third PIU module, transmittingthe ODU traffic from the M PIU ports of the third PIU module to theplurality of PIU modules other than the ith PIU port of the first PIUmodule and the ith PIU port of the second PIU module.
 12. The method ofclaim 1, further comprising: detecting a second fault condition of theith Ethernet fabric plane; and transmitting the ODU traffic from theplurality of PIU modules to the other Ethernet fabric planes.
 13. An OTNswitch comprising: an Ethernet fabric having a number M of Ethernetfabric planes, each of the M Ethernet fabric planes includes acorresponding Ethernet switch of M Ethernet switches; a plurality of PIUmodules each having M PIU ports including a first PIU module, wherein anith PIU port of each of the plurality of PIU modules is connected to theith Ethernet switch of the ith Ethernet fabric plane of the Ethernetfabric, and wherein a variable i having a value ranging from 1 to M todenote the ith Ethernet fabric plane of the M Ethernet fabric planes,the ith Ethernet switch of the M Ethernet switches, and the ith PIU portof the M PIU ports, wherein M is greater than one, the first PIU moduleto detect a fault condition associated with the ith PIU port of thefirst PIU module on the ith Ethernet fabric plane; and the OTN switch totransmit the fault condition to stop transmission of ODU traffic fromthe plurality of PIU modules to the ith PIU port of the first PIUmodule.
 14. The OTN switch of claim 13, further comprising: a second PIUmodule of the plurality of PIU modules to detect a second faultcondition associated with the ith PIU port of the second PIU module onthe ith Ethernet fabric plane; and the OTN switch to transmit the secondfault condition to stop the transmission of the ODU traffic from theplurality of PIU modules to the ith PIU port of the second PIU module.15. The OTN switch of claim 13, further comprising: the OTN switch,after receiving the fault condition and a second fault conditionassociated with the ith PIU port of a second PIU module of the pluralityof PIU modules on the ith Ethernet fabric plane, to: stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the second PIU module; and stop the transmission of theODU traffic from the plurality of PIU modules to the ith PIU port of thefirst PIU module and the ith PIU port of the second PIU module.
 16. TheOTN switch of claim 13, further comprising: the OTN switch, prior to thetransmission of the fault condition to stop the transmission of the ODUtraffic from the plurality of PIU modules to the ith PIU port of thefirst PIU module and the transmission of a second fault conditionassociated with the ith PIU port of a second PIU module of the pluralityof PIU modules on the ith Ethernet fabric plane to stop the transmissionof the ODU traffic from the plurality of PIU modules to the ith PIU portof the second PIU module, to stop the transmission of the ODU trafficfrom the M PIU ports of the first PIU module and the M PIU ports of thesecond PIU module; and the OTN switch, after expiration of a delayassociated with the first PIU module and the second PIU module, totransmit the second fault condition to stop the transmission of the ODUtraffic from the plurality of PIU modules to the ith PIU port of thesecond PIU module.
 17. The OTN switch of claim 16, further comprising:the OTN switch, after expiration of a second delay associated with thefirst PIU module and the second PIU module, to: transmit the ODU trafficfrom the M PIU ports of the first PIU module other than the ith PIU portof the first PIU module; and transmit the ODU traffic from the M PIUports of the second PIU module other than the ith PIU port of the secondPIU module, wherein the expiration of the second delay is after theexpiration of the delay.
 18. The OTN switch of claim 13, furthercomprising: a third PIU module of the plurality of PIU modules; the OTNswitch, prior to the transmission of the fault condition to stop thetransmission of the ODU traffic from the plurality of PIU modules to theith PIU port of the first PIU module and the transmission of a secondfault condition associated with the ith PIU port of a second PIU moduleof the plurality of PIU modules on the ith Ethernet fabric plane to stopthe transmission of the ODU traffic from the plurality of PIU modules tothe ith PIU port of the second PIU module, to: stop the transmission ofthe ODU traffic from the M PIU ports of the first PIU module and the MPIU ports of the second PIU module; and receive a third fault conditionassociated with the jth PIU port of the third PIU module on the ithEthernet fabric plane to stop the transmission of the ODU traffic fromthe plurality of PIU modules to the jth PIU port of the third PIUmodule; and the OTN switch, after expiration of a second delayassociated with the first PIU module and the second PIU module, to:transmit the second fault condition to stop the transmission of the ODUtraffic from the plurality of PIU modules to the ith PIU port of thesecond PIU module; and transmit the third fault condition to stop thetransmission of the ODU traffic from the plurality of PIU modules to thejth PIU port of the third PIU module.
 19. The OTN switch of claim 18,further comprising: the OTN switch, after expiration of a third delayassociated with the first PIU module and the second PIU module, totransmit the ODU traffic from the M PIU ports of the first PIU module,other than the ith PIU port of the first PIU module, and the M PIU portsof the second PIU module, other than the ith PIU port of the second PIUmodule, to the plurality of PIU modules other than the ith PIU port ofthe first PIU module, the ith PIU port of the second PIU module, and thejth PIU port of the third PIU module, wherein the expiration of thethird delay is after the expiration of the second delay.
 20. The OTNswitch of claim 13, further comprising: a second PIU module of theplurality of PIU modules; a third PIU module of the plurality of PIUmodules, after receiving the fault condition and a second faultcondition associated with the ith PIU port of the second PIU module onthe ith Ethernet fabric plane to stop the transmission of the ODUtraffic from the plurality of PIU modules to the ith PIU port of thesecond PIU module, to stop the transmission of the ODU traffic from theM PIU ports of the third PIU module; and the third PIU module, afterexpiration of a delay associated with the third PIU module, to transmitthe ODU traffic from the M PIU ports of the third PIU module to theplurality of PIU modules other than the ith PIU port of the first PIUmodule and the ith PIU port of the second PIU module.